March 2023 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

Re: Ceph Mgr/Dashboard Python depedencies: a new approach
by Casey Bodley 14 Jul '23

14 Jul '23

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

4 10

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 04 Jul '23

04 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

ln: failed to create hard link 'file name': Read-only file system
by Frank Schilder 26 May '23

26 May '23

Hi all, on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm un-taring a larger package (1.2G) and after some time I get these errors: ln: failed to create hard link 'file name': Read-only file system The strange thing is that this seems only temporary. When I used "ln src dst" for manual testing, the command failed as above. However, after that I tried "ln -v src dst" and this command created the hard link with exactly the same path arguments. During the period when the error occurs, I can't see any FS in read-only mode, neither on the NFS client nor the NFS server. Funny thing is that file creation and write still works, its only the hard-link creation that fails. For details, the set-up is: file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server other server: mount /shares/path as NFS More precisely, on the file-server: fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0 exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP On the host at DEST-IP: fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0 Both, the file server and the client server are virtual machines. The file server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64). When I change the NFS export from "async" to "sync" everything works. However, that's a rather bad workaround and not a solution. Although this looks like an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a race with scheduling and executing operations on the ceph-fs kernel mount. Has anyone seen something like that? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 13

Frequent calling monitor election
by Frank Schilder 04 May '23

04 May '23

Hi all, our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting: 2/9/23 4:42:30 PM[INF]overall HEALTH_OK 2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:40:00 PM[INF]overall HEALTH_OK 2/9/23 4:30:00 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:04 PM[INF]overall HEALTH_OK 2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:20:00 PM[INF]overall HEALTH_OK 2/9/23 4:10:00 PM[INF]overall HEALTH_OK 2/9/23 4:00:00 PM[INF]overall HEALTH_OK 2/9/23 3:50:00 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again? Thanks for any help! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 9

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

deploying Ceph using FQDN for MON / MDS Services
by Lokendra Rathour 21 Apr '23

21 Apr '23

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

7 20

ceph/daemon stable tag
by Jonas Nemeikšis 15 Apr '23

15 Apr '23

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

2 6

quincy v17.2.6 QE Validation status
by Yuri Weinstein 07 Apr '23

07 Apr '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59070#note-1 Release Notes - TBD The reruns were in the queue for 4 days because of some slowness issues. The core team (Neha, Radek, Laura, and others) are trying to narrow down the root cause. Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to test and merge at least one PR https://github.com/ceph/ceph/pull/50575 for the core) rgw - Casey fs - Venky (the fs suite has an unusually high amount of failed jobs, any reason to suspect it in the observed slowness?) orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - Laura is looking into failures upgrade/pacific-x - Laura is looking into failures upgrade/quincy-p2p - Laura is looking into failures client-upgrade-octopus-quincy-quincy - missing packages, Adam Kraitman is looking into it powercycle - Brad ceph-volume - needs a rerun on merged https://github.com/ceph/ceph-ansible/pull/7409 Please reply to this email with approval and/or trackers of known issues/PRs to address them. Also, share any findings or hypnosis about the slowness in the execution of the suite. Josh, Neha - gibba and LRC upgrades pending major suites approvals. RC release - pending major suites approvals. Thx YuriW

14 27

Cephadm - Error ENOENT: Module not found
by elia.oggian＠cscs.ch 06 Apr '23

06 Apr '23

Hello, After a successful upgrade of a Ceph cluster from 16.2.7 to 16.2.11, I needed to downgrade it back to 16.2.7 as I found an issue with the new version. I expected that running the downgrade with:`ceph orch upgrade start --ceph-version 16.2.7` should have worked fine. However, it blocked right after the downgrade of the first MGR daemon. In fact, the downgraded daemon is not able to use the cephadm module anymore. Any `ceph orch` command fails with the following error: ``` $ ceph orch ps Error ENOENT: Module not found ``` And the downgrade process is therefore blocked. These are the logs of the MGR when issuing the command: ``` Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.557+0000 7f828fe8c700 0 log_channel(audit) log [DBG] : from='client.3136173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.list_daemons(*(None, None), **{'daemon_id': None, 'host': None, 'refresh': False}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.get_feature_set(*(), **{}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 mgr.server reply reply (2) No such file or directory Module not found ``` Other interesting MGR logs are: ``` 2023-03-28T11:05:59.519+0000 7fcd16314700 4 mgr get_store get_store key: mgr/cephadm/upgrade_state 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Failed to construct class in 'cephadm' 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Traceback (most recent call last): e "/usr/share/ceph/mgr/cephadm/module.py", line 450, in __init__ elf.upgrade = CephadmUpgrade(self) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 111, in __init__ elf.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 92, in from_json eturn cls(**c) rror: __init__() got an unexpected keyword argument 'daemon_types' 2023-03-28T11:05:59.521+0000 7fcd16314700 -1 mgr operator() Failed to run module in active mode ('cephadm') ``` Which seem to relate to the new feature of staggered upgrades. Please note that before, everything was working fine with version 16.2.7. I am currently stuck in this situation with only one MGR daemon on version 16.2.11 which is the only one still working fine: ``` [root@astano01 ~]# ceph orch ps | grep mgr mgr.astano02.mzmewn astano02 *:8443,9283 running (5d) 43s ago 2y 455M - 16.2.11 7a63bce27215 e2d7806acf16 mgr.astano03.qtzccn astano03 *:8443,9283 running (3m) 22s ago 95m 383M - 16.2.7 463ec4b1fdc0 cc0d88864fa1 ``` Does anyone already faced this issue or knows how can I make the 16.2.7 MGR load the cephadm module correctly? Thanks in advance for any help!

2 2

Misplaced objects greater than 100%
by Johan Hattne 06 Apr '23

06 Apr '23

Dear all; Up until a few hours ago, I had a seemingly normally-behaving cluster (Quincy, 17.2.5) with 36 OSDs, evenly distributed across 3 of its 6 nodes. The cluster is only used for CephFS and the only non-standard configuration I can think of is that I had 2 active MDSs, but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB (all OSD hosts have 256 G of RAM) at some point in the past. Then I rebooted one of the OSD nodes. The rebooted node held one of the active MDSs. Now the node is back up: ceph -s says the cluster is healthy, but all PGs are in a active+clean+remapped state and 166.67% of the objects are misplaced (dashboard: -66.66% healthy). The data pool is a threefold replica with 5.4M object, the number of misplaced objects is reported as 27087410/16252446. The denominator in the ratio makes sense to me (16.2M / 3 = 5.4M), but the numerator does not. I also note that the ratio is *exactly* 5 / 3. The filesystem is still mounted and appears to be usable, but df reports it as 100% full; I suspect it would say 167% but that is capped somewhere. Any ideas about what is going on? Any suggestions for recovery? // Best wishes; Johan

3 6

CephFS thrashing through the page cache
by Ashu Pachauri 05 Apr '23

05 Apr '23

We have an internal use case where we back the storage of a proprietary database by a shared file system. We noticed something very odd when testing some workload with a local block device backed file system vs cephfs. We noticed that the amount of network IO done by cephfs is almost double compared to the IO done in case of a local file system backed by an attached block device. We also noticed that CephFS thrashes through the page cache very quickly compared to the amount of data being read and think that the two issues might be related. So, I wrote a simple test. 1. I wrote 10k files 400KB each using dd (approx 4 GB data). 2. I dropped the page cache completely. 3. I then read these files serially, again using dd. The page cache usage shot up to 39 GB for reading such a small amount of data. Following is the code used to repro this in bash: for i in $(seq 1 10000); do dd if=/dev/zero of=test_${i} bs=4k count=100 done sync; echo 1 > /proc/sys/vm/drop_caches for i in $(seq 1 10000); do dd if=test_${i} of=/dev/null bs=4k count=100 done The ceph version being used is: ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) The ceph configs being overriden: WHO MASK LEVEL OPTION VALUE RO mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/mode upmap mgr advanced mgr/dashboard/server_addr 127.0.0.1 * mgr advanced mgr/dashboard/server_port 8443 * mgr advanced mgr/dashboard/ssl false * mgr advanced mgr/prometheus/server_addr 0.0.0.0 * mgr advanced mgr/prometheus/server_port 9283 * osd advanced bluestore_compression_algorithm lz4 osd advanced bluestore_compression_mode aggressive osd advanced bluestore_throttle_bytes 536870912 osd advanced osd_max_backfills 3 osd advanced osd_op_num_threads_per_shard_ssd 8 * osd advanced osd_scrub_auto_repair true mds advanced client_oc false mds advanced client_readahead_max_bytes 4096 mds advanced client_readahead_max_periods 1 mds advanced client_readahead_min 0 mds basic mds_cache_memory_limit 21474836480 client advanced client_oc false client advanced client_readahead_max_bytes 4096 client advanced client_readahead_max_periods 1 client advanced client_readahead_min 0 client advanced fuse_disable_pagecache false The cephfs mount options (note that readahead was disabled for this test): /mnt/cephfs type ceph (rw,relatime,name=cephfs,secret=<hidden>,acl,rasize=0) Any help or pointers are appreciated; this is a major performance issue for us. Thanks and Regards, Ashu Pachauri

4 12

RGW can't create bucket
by Kamil Madac 05 Apr '23

05 Apr '23

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source -- Kamil Madac

3 6

compiling Nautilus for el9
by Marc 03 Apr '23

03 Apr '23

Is it possible to compile Nautilus for el9? Or maybe just the osd's?

2 4

How mClock profile calculation works, and IOPS
by Luis Domingues 03 Apr '23

03 Apr '23

Hi, I am reading reading some documentation about mClock and have two questions. First, about the IOPS. Are those IOPS disk IOPS or other kind of IOPS? And what the assumption of those? (Like block size, sequential or random reads/writes)? And the second question, How mClock calculates its profiles? I have my lab cluster running Quincy, and I have this parameters for mClock: "osd_mclock_max_capacity_iops_hdd": "450.000000", "osd_mclock_profile": "balanced", According to the documentation: https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#bala… I am expecting to have: "osd_mclock_scheduler_background_best_effort_lim": "999999", "osd_mclock_scheduler_background_best_effort_res": "90", "osd_mclock_scheduler_background_best_effort_wgt": "2", "osd_mclock_scheduler_background_recovery_lim": "675", "osd_mclock_scheduler_background_recovery_res": "180", "osd_mclock_scheduler_background_recovery_wgt": "1", "osd_mclock_scheduler_client_lim": "450", "osd_mclock_scheduler_client_res": "180", "osd_mclock_scheduler_client_wgt": "1", But what I get is: "osd_mclock_scheduler_background_best_effort_lim": "999999", "osd_mclock_scheduler_background_best_effort_res": "18", "osd_mclock_scheduler_background_best_effort_wgt": "2", "osd_mclock_scheduler_background_recovery_lim": "135", "osd_mclock_scheduler_background_recovery_res": "36", "osd_mclock_scheduler_background_recovery_wgt": "1", "osd_mclock_scheduler_client_lim": "90", "osd_mclock_scheduler_client_res": "36", "osd_mclock_scheduler_client_wgt": "1", Which seems very low according to what my disk seems to be able to handle. Is this calculation the expected one? Or did I miss something on how those profiles are populated? Luis Domingues Proton AG

2 5

Eccessive occupation of small OSDs
by Nicola Mori 03 Apr '23

03 Apr '23

Dear Ceph users, my cluster is made up of 10 old machines, with uneven number of disks and disk size. Essentially I have just one big data pool (6+2 erasure code, with host failure domain) for which I am currently experiencing a very poor available space (88 TB of which 40 TB occupied, as reported by df -h on hosts mounting the cephfs) compared to the raw one (196.5 TB). I have a total of 104 OSDs and 512 PGs for the pool; I cannot increment the PG number since the machines are old and with very low amount of RAM, and some of them are already overloaded. In this situation I'm seeing a high occupation of small OSDs (500 MB) with respect to bigger ones (2 and 4 TB) even if the weight is set equal to disk capacity (see below for ceph osd tree). For example OSD 9 is at 62% occupancy even with weight 0.5 and reweight 0.75, while the highest occupancy for 2 TB OSDs is 41% (OSD 18) and 4 TB OSDs is 23% (OSD 79). I guess this high occupancy for 500 MB OSDs combined with erasure code size and host failure domain might be the cause of the poor available space, could this be true? The upmap balancer is currently running but I don't know if and how much it could improve the situation. Any hint is greatly appreciated, thanks. Nicola # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 196.47754 root default -7 14.55518 host aka 4 hdd 1.81940 osd.4 up 1.00000 1.00000 11 hdd 1.81940 osd.11 up 1.00000 1.00000 18 hdd 1.81940 osd.18 up 1.00000 1.00000 26 hdd 1.81940 osd.26 up 1.00000 1.00000 32 hdd 1.81940 osd.32 up 1.00000 1.00000 41 hdd 1.81940 osd.41 up 1.00000 1.00000 48 hdd 1.81940 osd.48 up 1.00000 1.00000 55 hdd 1.81940 osd.55 up 1.00000 1.00000 -3 14.55518 host balin 0 hdd 1.81940 osd.0 up 1.00000 1.00000 8 hdd 1.81940 osd.8 up 1.00000 1.00000 15 hdd 1.81940 osd.15 up 1.00000 1.00000 22 hdd 1.81940 osd.22 up 1.00000 1.00000 29 hdd 1.81940 osd.29 up 1.00000 1.00000 34 hdd 1.81940 osd.34 up 1.00000 1.00000 43 hdd 1.81940 osd.43 up 1.00000 1.00000 49 hdd 1.81940 osd.49 up 1.00000 1.00000 -13 29.10950 host bifur 3 hdd 3.63869 osd.3 up 1.00000 1.00000 14 hdd 3.63869 osd.14 up 1.00000 1.00000 27 hdd 3.63869 osd.27 up 1.00000 1.00000 37 hdd 3.63869 osd.37 up 1.00000 1.00000 50 hdd 3.63869 osd.50 up 1.00000 1.00000 59 hdd 3.63869 osd.59 up 1.00000 1.00000 64 hdd 3.63869 osd.64 up 1.00000 1.00000 69 hdd 3.63869 osd.69 up 1.00000 1.00000 -17 29.10950 host bofur 2 hdd 3.63869 osd.2 up 1.00000 1.00000 21 hdd 3.63869 osd.21 up 1.00000 1.00000 39 hdd 3.63869 osd.39 up 1.00000 1.00000 57 hdd 3.63869 osd.57 up 1.00000 1.00000 66 hdd 3.63869 osd.66 up 1.00000 1.00000 72 hdd 3.63869 osd.72 up 1.00000 1.00000 76 hdd 3.63869 osd.76 up 1.00000 1.00000 79 hdd 3.63869 osd.79 up 1.00000 1.00000 -21 29.10376 host dwalin 88 hdd 1.81898 osd.88 up 1.00000 1.00000 89 hdd 1.81898 osd.89 up 1.00000 1.00000 90 hdd 1.81898 osd.90 up 1.00000 1.00000 91 hdd 1.81898 osd.91 up 1.00000 1.00000 92 hdd 1.81898 osd.92 up 1.00000 1.00000 93 hdd 1.81898 osd.93 up 1.00000 1.00000 94 hdd 1.81898 osd.94 up 1.00000 1.00000 95 hdd 1.81898 osd.95 up 1.00000 1.00000 96 hdd 1.81898 osd.96 up 1.00000 1.00000 97 hdd 1.81898 osd.97 up 1.00000 1.00000 98 hdd 1.81898 osd.98 up 1.00000 1.00000 99 hdd 1.81898 osd.99 up 1.00000 1.00000 100 hdd 1.81898 osd.100 up 1.00000 1.00000 101 hdd 1.81898 osd.101 up 1.00000 1.00000 102 hdd 1.81898 osd.102 up 1.00000 1.00000 103 hdd 1.81898 osd.103 up 1.00000 1.00000 -9 14.55518 host ogion 7 hdd 1.81940 osd.7 up 1.00000 1.00000 16 hdd 1.81940 osd.16 up 1.00000 1.00000 23 hdd 1.81940 osd.23 up 1.00000 1.00000 33 hdd 1.81940 osd.33 up 1.00000 1.00000 40 hdd 1.81940 osd.40 up 1.00000 1.00000 47 hdd 1.81940 osd.47 up 1.00000 1.00000 54 hdd 1.81940 osd.54 up 1.00000 1.00000 61 hdd 1.81940 osd.61 up 1.00000 1.00000 -19 14.55518 host prestno 81 hdd 1.81940 osd.81 up 1.00000 1.00000 82 hdd 1.81940 osd.82 up 1.00000 1.00000 83 hdd 1.81940 osd.83 up 1.00000 1.00000 84 hdd 1.81940 osd.84 up 1.00000 1.00000 85 hdd 1.81940 osd.85 up 1.00000 1.00000 86 hdd 1.81940 osd.86 up 1.00000 1.00000 87 hdd 1.81940 osd.87 up 1.00000 1.00000 104 hdd 1.81940 osd.104 up 1.00000 1.00000 -15 29.10376 host remolo 6 hdd 1.81897 osd.6 up 1.00000 1.00000 12 hdd 1.81897 osd.12 up 1.00000 1.00000 19 hdd 1.81897 osd.19 up 1.00000 1.00000 28 hdd 1.81897 osd.28 up 1.00000 1.00000 35 hdd 1.81897 osd.35 up 1.00000 1.00000 44 hdd 1.81897 osd.44 up 1.00000 1.00000 52 hdd 1.81897 osd.52 up 1.00000 1.00000 58 hdd 1.81897 osd.58 up 1.00000 1.00000 63 hdd 1.81897 osd.63 up 1.00000 1.00000 67 hdd 1.81897 osd.67 up 1.00000 1.00000 71 hdd 1.81897 osd.71 up 1.00000 1.00000 73 hdd 1.81897 osd.73 up 1.00000 1.00000 74 hdd 1.81897 osd.74 up 1.00000 1.00000 75 hdd 1.81897 osd.75 up 1.00000 1.00000 77 hdd 1.81897 osd.77 up 1.00000 1.00000 78 hdd 1.81897 osd.78 up 1.00000 1.00000 -5 14.55518 host rokanan 1 hdd 1.81940 osd.1 up 1.00000 1.00000 10 hdd 1.81940 osd.10 up 1.00000 1.00000 17 hdd 1.81940 osd.17 up 1.00000 1.00000 24 hdd 1.81940 osd.24 up 1.00000 1.00000 31 hdd 1.81940 osd.31 up 1.00000 1.00000 38 hdd 1.81940 osd.38 up 1.00000 1.00000 46 hdd 1.81940 osd.46 up 1.00000 1.00000 53 hdd 1.81940 osd.53 up 1.00000 1.00000 -11 7.27515 host romolo 5 hdd 0.45470 osd.5 up 1.00000 1.00000 9 hdd 0.45470 osd.9 up 0.75000 1.00000 13 hdd 0.45470 osd.13 up 1.00000 1.00000 20 hdd 0.45470 osd.20 up 0.95000 1.00000 25 hdd 0.45470 osd.25 up 0.75000 1.00000 30 hdd 0.45470 osd.30 up 1.00000 1.00000 36 hdd 0.45470 osd.36 up 1.00000 1.00000 42 hdd 0.45470 osd.42 up 1.00000 1.00000 45 hdd 0.45470 osd.45 up 0.85004 1.00000 51 hdd 0.45470 osd.51 up 0.89999 1.00000 56 hdd 0.45470 osd.56 up 1.00000 1.00000 60 hdd 0.45470 osd.60 up 1.00000 1.00000 62 hdd 0.45470 osd.62 up 1.00000 1.00000 65 hdd 0.45470 osd.65 up 0.85004 1.00000 68 hdd 0.45470 osd.68 up 1.00000 1.00000 70 hdd 0.45470 osd.70 up 1.00000 1.00000

4 5

monitoring apply_latency / commit_latency ?
by Matthias Ferdinand 02 Apr '23

02 Apr '23

Hi, I would like to understand how the per-OSD data from "ceph osd perf" (i.e. apply_latency, commit_latency) is generated. So far I couldn't find documentation on this. "ceph osd perf" output is nice for a quick glimpse, but is not very well suited for graphing. Output values are from the most recent 5s-averages apparently. With "ceph daemon osd.X perf dump" OTOH, you get quite a lot of latency metrics, while it is just not obvious to me how they aggregate into apply_latency and commit_latency. Or some comparably easy read latency metric (something that is missing completely in "ceph osd perf"). Can somebody shed some light on this? Regards Matthias

2 4

avg apply latency went up after update from octopus to pacific
by Boris Behrens 02 Apr '23

02 Apr '23

Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms. All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value? Cheers Boris

10 27

Failing to create monitor in a working cluster.
by pmestre＠gmail.com 01 Apr '23

01 Apr '23

Hello, i've been running a 3 node proxmox cluster with 4 ceph osd for 3 years as a production cluster. As a test for trying to move ceph cluster network, i destroyed one of the 3 working monitors and tried to recreate it. After destroying it, the new monitor refuses to join the cluster, even in the old network. I've tried all steps in documentation "Troubleshooting monitors" section. New monitor has this config extracted from ceph --admin-daemon file.asok { "name": "n3ceph", "rank": -1, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "2449958197560098820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "0", "quorum_mon": [] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false } The quorum mon stat is as follows: { "name": "node1", "rank": 0, "state": "leader", "election_epoch": 340, "quorum": [ 0, 1 ], "quorum_age": 13090, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138314316775423", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 5 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 1 }, { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 12 } ], "mgr": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false I tried to get a debug log with ceph daemon mon.n3ceph config set debug_mon 10/10 and restarting the service, but the ceph log file stoped working after i tried that setting. journalctl -u tells me: mar 31 17:35:22 node3 ceph-mon[240916]: 2023-03-31T17:35:22.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:27 node3 ceph-mon[240916]: 2023-03-31T17:35:27.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:32 node3 ceph-mon[240916]: 2023-03-31T17:35:32.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200). Any ideas? Cluster is running fine with two monitors, but a reboot in one of the nodes might be a big problem. Kind regards and many thanks.

2 1

Ceph Failure and OSD Node Stuck Incident
by petersun＠raksmart.com 31 Mar '23

31 Mar '23

We encountered a Ceph failure where the system became unresponsive with no IOPS or throughput after encountering a failed node. Upon investigation, it appears that the OSD process on one of the Ceph storage nodes is stuck, but ping is still responsive. However, during the failure, Ceph was unable to recognize the problematic node, which resulted in all other OSDs in the cluster experiencing slow operations and no IOPS in the cluster at all. Here's the timeline of the incident: - At 10:40, an alert is triggered, indicating a problem with the OSD. - After the alert, Ceph becomes unresponsive with no IOPS or throughput. - At 11:26, an engineer discovers that there is a gradual OSD failure, with 6 out of 12 OSDs on the node being down. - At 11:46, the Ceph engineer is unable to SSH into the faulty node and attempts a soft restart, but the "smartmontools" process is stuck while shutting down the server. Ping works during this time. - After waiting for about one or two minutes, a hard restart is attempted for the server. - At 11:57, after the Ceph node starts normally, service resumes as usual, indicating that the issue has been resolved. Here is some basic information about our services: - `Mon: 5 daemons, quorum host001, host002, host003, host004, host005 (age 4w)` - `Mgr: host005 (active, since 4w), standbys: host001, host002, host003, host004` - `Osd: 218 osds: 218 up (since 22h), 218 in (since 22h)` We have a cluster with 19 nodes, including 15 SSD nodes and 4 HDD nodes. In total, there are 218 OSDs. The SSD nodes have 11 OSDs with Samsung EVO 870 SSD and each drive DB/WAL by 1.6T NVME drive. We are using Ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable). Here is the health check detail: [root@node21 ~]# ceph health detail HEALTH_WARN 1 osds down; Reduced data availability: 12 pgs inactive, 12 pgs peering; Degraded data redundancy: 272273/43967625 objects degraded (0.619%), 88 pgs degraded, 5 pgs undersized; 18192 slow ops, oldest one blocked for 3730 sec, daemons [osd.0,osd.1,osd.101,osd.103,osd.107,osd.108,osd.109,osd.11,osd.111,osd.112]... have slow ops. [WRN] OSD_DOWN: 1 osds down osd.174 (root=default,host=hkhost031) is down [WRN] PG_AVAILABILITY: Reduced data availability: 12 pgs inactive, 12 pgs peering pg 2.dc is stuck peering for 49m, current state peering, last acting [87,95,172] pg 2.e2 is stuck peering for 15m, current state peering, last acting [51,177,97] ...... pg 2.f7e is active+undersized+degraded, acting [10,214] pg 2.f84 is active+undersized+degraded, acting [91,52] [WRN] SLOW_OPS: 18192 slow ops, oldest one blocked for 3730 sec, daemons [osd.0,osd.1,osd.101,osd.103,osd.107,osd.108,osd.109,osd.11,osd.111,osd.112]... have slow ops. I have the following questions: 1. Why couldn't Ceph detect the faulty node and automatically abandon its resources? Can anyone provide more troubleshooting guidance for this case? 2. What is Ceph's detection mechanism and where can I find related information? All of our production cloud machines were affected and suspended. If RBD is unstable, we cannot continue to use Ceph technology for our RBD source. 3. Did we miss any patches or bug fixes? 4. Is there anyone who can suggest improvements and how we can quickly detect and avoid similar issues in the future?

5 4

./install-deps.sh takes several hours
by Arvid Picciani 31 Mar '23

31 Mar '23

Hi again, something is very wrong with my hardware it seems and i'm slowly turning insane. I'm trying to debug why ceph has incredibly poor performance for us. we've got - 3 EPYC 7713 dual-cpu systems - datacenter nvme drives (3GB/s top) - 100G infiniband ceph does 800MB/s read max, CPU is idle, network is idle, I/O nowhere near saturation now while hunting for queues, i wanted to recompile ceph and poke around abit, but ./install-deps.sh is now running for 6 hours with the exact same problem. CPU is idle, network is idle, zero I/O what am i doing wrong? is there another computer resource we're bottlenecked on that i just dont know about? -- +4916093821054

1 0

ceph orch ps mon, mgr, osd shows <unknown> for version, image and container id
by anantha.adiga＠intel.com 31 Mar '23

31 Mar '23

Hi , Why is ceph orch ps showing ,unknown version, image and container id ? root@a002s002:~# cephadm shell ceph mon versions Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 } root@a002s002:~# cephadm shell ceph mgr versions Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 } root@a002s002:~# cephadm shell ceph orch ps --daemon-type mgr Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mgr.a002s002 a002s002 running 4m ago 11M - - <unknown> <unknown> mgr.a002s003 a002s003 running 87s ago 11M - - <unknown> <unknown> mgr.a002s004 a002s004 running 4m ago 11M - - <unknown> <unknown> root@a002s002:~# cephadm shell ceph orch ps --daemon-type mon Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a002s002 a002s002 running 4m ago 11M - 2048M <unknown> <unknown> <unknown> mon.a002s003 a002s003 running 95s ago 11M - 2048M <unknown> <unknown> <unknown> mon.a002s004 a002s004 running (4w) 4m ago 5M 1172M 2048M 16.2.5 6e73176320aa d38b94e00d28 root@a002s002:~# cephadm shell ceph orch ps --daemon-type osd Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.0 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.1 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.10 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.11 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.12 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.13 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.14 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.15 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.16 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.17 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.18 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.19 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.2 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.20 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.21 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown>

3 4

how to set block.db size
by li.xuehai＠99cloud.net 31 Mar '23

31 Mar '23

hi, how to understand ``Improvements in releases beginning with Nautilus 14.2.12 and Octopus 15.2.6 enable better utilization of arbitrary DB device sizes, and the Pacific release brings experimental dynamic level support. `` in the document ``https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#sizing`` is there a related patch??

1 0

Upgrade from 16.2.7. to 16.2.11 failing on OSDs
by Lo Re Giuseppe 31 Mar '23

31 Mar '23

Dear all, On one of our clusters I started the upgrade process from 16.2.7 to 16.2.11. Mon and mgr and crash processes were done easily/quickly, then at the first attempt of upgrading a OSD container the upgrade process stopped because of the OSD process is not able to start after the upgrade. Does anyone have any hint on how to unblock the upgrade? Some details below: Regards, Giuseppe I started the upgrade process with the cephadm command: “”” [root@naret-monitor01 ~]# ceph orch upgrade start --ceph-version 16.2.11 Initiating upgrade to quay.io/ceph/ceph:v16.2.11 “”” After a short time: “”” [root@naret-monitor01 ~]# ceph orch upgrade status { "target_image": quay.io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add<mailto:quay.io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add>, "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mon", "mgr" ], "progress": "64/2039 daemons upgraded", "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.4 on host naret-osd01 failed.", "is_paused": true } “”” The ceph health command reports: “”” [root@naret-monitor01 ~]# ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 2654362/6721382840 objects degraded (0.039%), 14 pgs degraded, 14 pgs undersized; Upgrading daemon osd.4 on host naret-osd01 failed. [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.22 on naret-osd01 is in error state [WRN] OSD_DOWN: 1 osds down osd.4 (root=default,host=naret-osd01) is down [WRN] PG_DEGRADED: Degraded data redundancy: 2654362/6721382840 objects degraded (0.039%), 14 pgs degraded, 14 pgs undersized pg 28.88 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1373,1337,1508,852,2147483647,483] pg 28.528 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1063,793,2147483647,931,338,1777] pg 28.594 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1208,891,1651,364,2147483647,53] pg 28.8b4 is stuck undersized for 6m, current state active+undersized+degraded, last acting [521,1273,1238,138,1539,2147483647] pg 28.a90 is stuck undersized for 6m, current state active+undersized+degraded, last acting [237,1665,1836,2147483647,192,1410] pg 28.ad6 is stuck undersized for 6m, current state active+undersized+degraded, last acting [870,466,350,885,1601,2147483647] pg 28.b34 is stuck undersized for 6m, current state active+undersized+degraded, last acting [920,1596,2147483647,115,201,941] pg 28.c14 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1389,424,2147483647,268,1646,632] pg 28.dba is stuck undersized for 6m, current state active+undersized+degraded, last acting [1099,561,2147483647,1806,1874,1145] pg 28.ee2 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1621,1904,1044,2147483647,1545,722] pg 29.163 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1883,2147483647,1509,1697,1187,235] pg 29.1c1 is stuck undersized for 6m, current state active+undersized+degraded, last acting [122,1226,962,1254,1215,2147483647] pg 29.254 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1782,1839,1545,412,196,2147483647] pg 29.2a1 is stuck undersized for 6m, current state active+undersized+degraded, last acting [370,2147483647,575,1423,1755,446] [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.4 on host naret-osd01 failed. Upgrade daemon: osd.4: cephadm exited with an error code: 1, stderr:Redeploy daemon osd.4 ... Non-zero exit code 1 from systemctl start ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4> systemctl: stderr Job for ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> failed because a timeout was exceeded. systemctl: stderr See "systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>" and "journalctl -xe" for details. Traceback (most recent call last): File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 9248, in <module> main() File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 9236, in main r = ctx.func(ctx) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 1990, in _default_image return func(ctx) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 5041, in command_deploy ports=daemon_ports) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 2952, in deploy_daemon c, osd_fsid=osd_fsid, ports=ports) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 3197, in deploy_daemon_units call_throws(ctx, ['systemctl', 'start', unit_name]) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 1657, in call_throws raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') RuntimeError: Failed command: systemctl start ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4>: Job for ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> failed because a timeout was exceeded. See "systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>" and "journalctl -xe" for details. “”” On the OSD server we have: “”” [root@naret-osd01 ~]# uname -a Linux naret-osd01 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Wed Dec 14 16:00:01 EST 2022 x86_64 x86_64 x86_64 GNU/Linux [root@naret-osd01 ~]# podman -v podman version 4.2.0 [root@naret-osd01 ~]# ceph -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) [root@naret-osd01 ~]# cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.7 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.7" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL=https://www.redhat.com/ DOCUMENTATION_URL=https://access.redhat.com/documentation/red_hat_enterpris… BUG_REPORT_URL=https://bugzilla.redhat.com/ REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.7 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.7" “”” Systemctl says: “”” systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> … ● ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> - Ceph osd.4 for 63334166-d991-11eb-99de-40a6b72108d0 Loaded: loaded (/etc/systemd/system/ceph-63334166-d991-11eb-99de-40a6b72108d0@.service<mailto:/etc/systemd/system/ceph-63334166-d991-11eb-99de-40a6b72108d0@.service>; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Mon 2023-03-27 15:34:29 CEST; 6min ago Process: 730621 ExecStopPost=/bin/rm -f /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-pid> /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-cid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-cid> (code=exited, status=0/SUCCESS) Process: 730209 ExecStopPost=/bin/bash /var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/osd.4/unit.poststop (code=exited, status=0/SUCCESS) Process: 710355 ExecStartPre=/bin/rm -f /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-pid> /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-cid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-cid> (code=exited, status=0/SUCCESS) Main PID: 23025 (code=exited, status=0/SUCCESS) Tasks: 62 (limit: 1647878) Memory: 961.8M CGroup: /system.slice/system-ceph\x2d63334166\x2dd991\x2d11eb\x2d99de\x2d40a6b72108d0.slice/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service ├─libpod-payload-b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e │ └─754976 /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug └─runtime └─754965 /usr/bin/conmon --api-version 1 -c b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e -u b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata -p /run/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata/pidfile -n ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4 --exit-dir /run/libpod/exits --full-attach -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata/oci-log --conmon-pidfile /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-pid> --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e6ae1700 1 osd.4 pg_epoch: 821628 pg[28.dbas2( v 821618'4657799 (819107'4647770,821618'4657799] local-lis/les=749842/749843 n=239290 ec=130297/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=7.751406670s) [1099,561,4,1806,1874,1145]p1099(0) r=2 lpr=821628 pi=[749842,821628)/1 crt=821618'4657799 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.039081573s@ mbc={} ps=[4~6]] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e8ae5700 1 osd.4 pg_epoch: 821628 pg[29.163s1( v 821572'139334 (776804'129273,821572'139334] local-lis/les=749851/749852 n=65683 ec=130801/130801 lis/c=821623/749851 les/c/f=821624/749852/0 sis=821628 pruub=8.023463249s) [1883,4,1509,1697,1187,235]p1883(0) r=1 lpr=821628 pi=[749851,821628)/1 crt=821572'139334 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.311203003s@ mbc={}] start_peering_interval up [1883,4,1509,1697,1187,235] -> [1883,4,1509,1697,1187,235], acting [1883,2147483647,1509,1697,1187,235] -> [1883,4,1509,1697,1187,235], acting_primary 1883(0) -> 1883, up_primary 1883(0) -> 1883, role -1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[29.2a1s1( v 821500'140649 (776804'130601,821500'140649] local-lis/les=749849/749850 n=65848 ec=130801/130801 lis/c=821623/749849 les/c/f=821624/749850/0 sis=821628 pruub=7.845988274s) [370,4,575,1423,1755,446]p370(0) r=1 lpr=821628 pi=[749849,821628)/1 crt=821500'140649 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.133728981s@ mbc={}] start_peering_interval up [370,4,575,1423,1755,446] -> [370,4,575,1423,1755,446], acting [370,2147483647,575,1423,1755,446] -> [370,4,575,1423,1755,446], acting_primary 370(0) -> 370, up_primary 370(0) -> 370, role -1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e8ae5700 1 osd.4 pg_epoch: 821628 pg[29.163s1( v 821572'139334 (776804'129273,821572'139334] local-lis/les=749851/749852 n=65683 ec=130801/130801 lis/c=821623/749851 les/c/f=821624/749852/0 sis=821628 pruub=8.023443222s) [1883,4,1509,1697,1187,235]p1883(0) r=1 lpr=821628 pi=[749851,821628)/1 crt=821572'139334 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.311203003s@ mbc={}] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[29.2a1s1( v 821500'140649 (776804'130601,821500'140649] local-lis/les=749849/749850 n=65848 ec=130801/130801 lis/c=821623/749849 les/c/f=821624/749850/0 sis=821628 pruub=7.845966339s) [370,4,575,1423,1755,446]p370(0) r=1 lpr=821628 pi=[749849,821628)/1 crt=821500'140649 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.133728981s@ mbc={}] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[28.8b4s5( v 821618'2906095 (817032'2896088,821618'2906095] local-lis/les=749842/749843 n=239377 ec=130295/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=8.158309937s) [521,1273,1238,138,1539,4]p521(0) r=5 lpr=821628 pi=[749842,821628)/1 crt=821618'2906095 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.446221352s@ mbc={} ps=[4~6]] start_peering_interval up [521,1273,1238,138,1539,4] -> [521,1273,1238,138,1539,4], acting [521,1273,1238,138,1539,2147483647] -> [521,1273,1238,138,1539,4], acting_primary 521(0) -> 521, up_primary 521(0) -> 521, role -1 -> 5, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[28.8b4s5( v 821618'2906095 (817032'2896088,821618'2906095] local-lis/les=749842/749843 n=239377 ec=130295/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=8.158291817s) [521,1273,1238,138,1539,4]p521(0) r=5 lpr=821628 pi=[749842,821628)/1 crt=821618'2906095 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.446221352s@ mbc={} ps=[4~6]] state<Start>: transitioning to Stray Mar 27 15:39:36 naret-osd01 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>: Start request repeated too quickly. Mar 27 15:39:36 naret-osd01 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>: Failed with result 'timeout'. Mar 27 15:39:36 naret-osd01 systemd[1]: Failed to start Ceph osd.4 for 63334166-d991-11eb-99de-40a6b72108d0. “””

2 1

how ceph OSD bench works?
by Luis Domingues 31 Mar '23

31 Mar '23

Hi, I am currently testing some new disks, doing some benchmarks and stuff, and I would like to undertand how the OSD bench works. If I quicky explain our setup, we have a small ceph cluster, where our new disks are inserted. And we have some pools with no replication at all, and 1 PG only, up-mapped to those new disks. So I can do some benchmarks on them. The thing that is odd, is that doing some tests with fio tool, I have similar results on all disks, and doing the rados bench during 5 minutes as well. But the OSD bench at startup of the OSD, for mClock to configure osd_mclock_max_capacity_iops_hdd gives me a very big difference between disks. (600 vs 2200). I am running Pacific on this test cluster. Is there anywhere documentation of how this works? Or if anyone could explain that would be great. I did not found any documentation on how OSD benchmark works, only how to used it. But playing a little bit with it, it seems the results we get is highly dependent on the block sizes we use. Same for rados bench, results are dependent, at least on my tests, of the block size we use, which I found a little bit weird to be honest. And as mClock depends on that, it is impactful performance wise. On our cluster we can reach a lot better performances if we teak those values, instead of letting the cluster do proper measurements. And this looks to impact certain disk vendors more than others. Luis Domingues Proton AG

3 3

Call for Submissions IO500 ISC23
by IO500 Committee 30 Mar '23

30 Mar '23

Stabilization Period: Monday, April 3rd - Friday, April 14th, 2023 Submission Deadline: Tuesday, May 16st, 2023 AoE The IO500 is now accepting and encouraging submissions for the upcoming 12th semi-annual IO500 list, in conjunction with ISC23. Once again, we are also accepting submissions to the 10 Client Node Challenge to encourage the submission of small scale results. The new ranked lists will be announced at the ISC23 BoF [1]. We hope to see many new results. What's New 1. Creation of Production and Research Lists - Starting with ISC'22, we proposed a separation of the list into separate Production and Research lists. This better reflects the important distinction between storage systems that run in production environments and those that may use more experimental hardware and software configurations. At ISC23, we will formally create these two lists and users will be able to submit to either of the two lists (and their 10 client-node counterparts). Please see the requirements for each list on the IO500 rules page [3]. 2. New Submission Tool - There is now a new IO500 submission tool that improves the overall submission experience. Users can create accounts and then update and manage all of their submissions through that account. As part of this new tool, we have improved the submission fields that describe the hardware and software of the system under test. For reproducibility and analysis reasons, we now made the easily obtainable fields mandatory - data from storage servers are for users often difficult to obtain, therefore, most remain optional. As a new system, there may be quirks, please reach out on Slack or the mailing list if you see any issues. Further details will be released on the submission page [2]. 3. Reproducibility - Every submission will now receive a reproducibility score based upon the provided system details and the reproducibility questionnaire. This score will inform the community on the amount of details provided in the submission and the obtainability of the storage system. Further, this score will be used to evaluate if a submission is eligible for the Production list. 4. New Phases - We are continuing to evaluate the inclusion of optional test phases for additional key workloads - split easy/hard find phases, 4KB and 1MB random read/write phases, and concurrent metadata operations. This is called an extended run. At the moment, we collect the information to verify that additional phases do not significantly impact the results of a standard run and an extended run to facilitate comparisons between the existing and new benchmark phases. In a future release, we may include some or all of these results as part of the standard benchmark. The extended results are not currently included in the scoring of any ranked list. Background The benchmark suite is designed to be easy to run and the community has multiple active support channels to help with any questions. Please note that submissions of all sizes are welcome; the site has customizable sorting, so it is possible to submit on a small system and still get a very good per-client score, for example. Additionally, the list is about much more than just the raw rank; all submissions help the community by collecting and publishing a wider corpus of data. More details below. Following the success of the Top500 in collecting and analyzing historical trends in supercomputer technology and evolution, the IO500 was created in 2017, published its first list at SC17, and has grown continually since then. The need for such an initiative has long been known within High-Performance Computing; however, defining appropriate benchmarks has long been challenging. Despite this challenge, the community, after long and spirited discussion, finally reached consensus on a suite of benchmarks and a metric for resolving the scores into a single ranking. The multi-fold goals of the benchmark suite are as follows: 1. Maximizing simplicity in running the benchmark suite 2. Encouraging optimization and documentation of tuning parameters for performance 3. Allowing submitters to highlight their "hero run" performance numbers 4. Forcing submitters to simultaneously report performance for challenging IO patterns. Specifically, the benchmark suite includes a hero-run of both IOR and mdtest configured however possible to maximize performance and establish an upper-bound for performance. It also includes an IOR and mdtest run with highly prescribed parameters in an attempt to determine a lower performance bound. Finally, it includes a namespace search as this has been determined to be a highly sought-after feature in HPC storage systems that has historically not been well-measured. Submitters are encouraged to share their tuning insights for publication. The goals of the community are also multi-fold: 1. Gather historical data for the sake of analysis and to aid predictions of storage futures 2. Collect tuning information to share valuable performance optimizations across the community 3. Encourage vendors and designers to optimize for workloads beyond "hero runs" 4. Establish bounded expectations for users, procurers, and administrators The IO500 follows a two-staged approach. First, there will be a two-week stabilization period during which we encourage the community to verify that the benchmark runs properly on a variety of storage systems. During this period the benchmark may be updated based upon feedback from the community. The final benchmark will then be released. We expect that runs compliant with the rules made during the stabilization period will be valid as a final submission unless a significant defect is found. 10 Client Node I/O Challenge The 10 Client Node Challenge is conducted using the regular IO500 benchmark, however, with the rule that exactly 10 client nodes must be used to run the benchmark. You may use any shared storage with any number of servers. We will announce the results in the Production and Research lists as well as in separate derived lists. Birds-of-a-Feather Once again, we encourage you to submit [2] to join our community, and to attend the ISC23 BoF [1], where we will announce the new IO500 Production and Research lists and their 10 client node counterparts. The current list includes results from twenty different storage system types and 70 institutions. We hope that the upcoming list grows even more. [1] https://io500.org/pages/bof-isc23 [2] https://io500.org/submission [3] https://io500.org/rules-submission -- The IO500 Committee

1 0

OSD will not start - ceph_assert(r == q->second->file_map.end())
by Pat Vaughan 30 Mar '23

30 Mar '23

I have a cluster that I increased the the number of PGs on because the autoscaler wasn't working as expected. It's recovering the misplaced objects, but a OSD just failed, and refuses to come back up. The device is readable to the OS, and there are 2 other OSDs on the same node that are online. I looked online, but haven't found anything relevant. This is the end of the OSD log: -3> 2023-03-30T21:21:19.641+0000 7fcb026413c0 1 bluefs mount -2> 2023-03-30T21:21:19.641+0000 7fcb026413c0 1 bluefs _init_alloc shared, id 1, capacity 0x4affc00000, block size 0x10000 -1> 2023-03-30T21:21:19.673+0000 7fcb026413c0 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_replay(bool, bool)' thread 7fcb026413c0 time 2023-03-30T21:21:19.665811+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.3/rpm/el8/BUILD/ceph-17.2.3/src/os/bluestore/BlueFS.cc: 1419: FAILED ceph_assert(r == q->second->file_map.end()) ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x56525ddd8954] 2: /usr/bin/ceph-osd(+0x5d8b75) [0x56525ddd8b75] 3: (BlueFS::_replay(bool, bool)+0x599c) [0x56525e5590ec] 4: (BlueFS::mount()+0x120) [0x56525e559530] 5: (BlueStore::_open_bluefs(bool, bool)+0x94) [0x56525e4160b4] 6: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x6e1) [0x56525e417211] 7: (BlueStore::_open_db(bool, bool, bool)+0x159) [0x56525e449f69] 8: (BlueStore::_open_db_and_around(bool, bool)+0x2b4) [0x56525e493f14] 9: (BlueStore::_mount()+0x1ae) [0x56525e4970fe] 10: (OSD::init()+0x403) [0x56525df16f23] 11: main() 12: __libc_start_main() 13: _start() 0> 2023-03-30T21:21:19.681+0000 7fcb026413c0 -1 *** Caught signal (Aborted) ** in thread 7fcb026413c0 thread_name:ceph-osd ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fcb00844ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56525ddd89b2] 5: /usr/bin/ceph-osd(+0x5d8b75) [0x56525ddd8b75] 6: (BlueFS::_replay(bool, bool)+0x599c) [0x56525e5590ec] 7: (BlueFS::mount()+0x120) [0x56525e559530] 8: (BlueStore::_open_bluefs(bool, bool)+0x94) [0x56525e4160b4] 9: (BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x6e1) [0x56525e417211] 10: (BlueStore::_open_db(bool, bool, bool)+0x159) [0x56525e449f69] 11: (BlueStore::_open_db_and_around(bool, bool)+0x2b4) [0x56525e493f14] 12: (BlueStore::_mount()+0x1ae) [0x56525e4970fe] 13: (OSD::init()+0x403) [0x56525df16f23] 14: main() 15: __libc_start_main() 16: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_cleaner 0/ 5 seastore_lba 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 alienstore 1/ 5 mclock -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7fcafb03b700 / admin_socket 7fcafb83c700 / msgr-worker-2 7fcb026413c0 / ceph-osd max_recent 10000 max_new 10000 log_file /var/log/ceph/ceph-osd.5.log --- end dump of recent events --- I'd like to recover this OSD if possible. Does anyone have any suggestions?

1 0

Controlling the number of open files from ceph client
by bhattacharya.soumya.ou＠gmail.com 30 Mar '23

30 Mar '23

Hi Ceph Users, My goal is to control the number of files a ceph client can open to the backend ceph filesystem at once to control the metadata transaction load. In this experiment, I have a ceph client on version Quincy on a physical server. The fstab entry below shows the options with which the ceph filesystem is mounted on it. In particular, I used caps_max=1900 option with the intention to limit the total number of open files by this client at once to 1900 at the mount point /ourdisk/hpc_scratch. # cat /etc/fstab ... 10.251.0.30:6789,10.251.0.31:6789,10.251.0.32:6789,10.251.0.33:6789,10.251.0.40:6789:/volumes/hpc_scratch/scratch/525205e7-4f71-4383-89ce-53e2ec68d017 /ourdisk/hpc_scratch ceph fsc,caps_max=1900,name=oscer,secretfile=/etc/ceph/client.oscer.secret 0 0 To test if this worked, I used fio command to create 4000 files at once in a directory on Ceph at /ourdisk/hpc_scratch/soumya/fio_tests/client_c003. During the run, I looked at the caps file (copied below), and it shows the number of used caps is 4007. Given my limited knowledge, it seems to me that the client was able to open 4000 files at once. # head /sys/kernel/debug/ceph/d5d5b0aa-1867-11eb-9f4a-bc97e1724ff1.client483151133/caps total 5032 avail 1025 used 4007 reserved 0 min 1024 ino mds issued implemented -------------------------------------------------- 0x200136e01b6 0 pAsLsXsFs pAsLsXsFs 0x1 0 pAsLsXsFs pAsLsXsFs Is there any way I can control how many simultaneous files a ceph client can open (preferably from the client side, if not then from the ceph side but on an individual client)? If so, how can I check the number of files a client is opening at a given time? Thank you for your time, Soumya PS: I understand that the number of capabilities is not the number of open files, but that's the closest mount option I could find for this experiment.

1 0

17.2.6 RC available
by Yuri Weinstein 30 Mar '23

30 Mar '23

We are publishing a release candidate this time for users to try for testing only. Please note this RC had only limited testing. Full testing is being done now. Upgrading has been tested on some internal clusters, and the final upgrade of the longest-running cluster there is in progress. The branch name: https://github.com/ceph/ceph/tree/quincy-release https://shaman.ceph.com/builds/ceph/quincy-release/23eb3b2f0fc65087846571af… To install dev packages see https://docs.ceph.com/en/quincy/install/get-packages/#ceph-development-pack… The container build see - https://quay.ceph.io/ceph-ci/ceph:quincy-release Release notes: https://github.com/ceph/ceph/pull/50721 ***Don’t use this RC on production clusters!*** The goal is to give users time to test and give feedback on RC releases while our upstream long-running cluster also runs the same RC release during that time (period of one week). Please respond to this email to provide any feedback on issues found in this release. Thx YuriW

1 0

OSD down cause all OSD slow ops
by petersun＠raksmart.com 30 Mar '23

30 Mar '23

We experienced a Ceph failure causing the system to become unresponsive with no IOPS or throughput due to a problematic OSD process on one node. This resulted in slow operations and no IOPS for all other OSDs in the cluster. The incident timeline is as follows: Alert triggered for OSD problem. 6 out of 12 OSDs on the node were down. Soft restart attempted, but smartmontools process stuck while shutting down server. Hard restart attempted and service resumed as usual. Our Ceph cluster has 19 nodes, 218 OSDs, and is using version 15.2.17 octopus (stable). Questions: 1. What is Ceph's detection mechanism? Why couldn't Ceph detect the faulty node and automatically abandon its resources? 2. Did we miss any patches or bug fixes? 3. Suggestions for improvements to quickly detect and avoid similar issues in the future?

2 1

Re: Upgrade from 16.2.7. to 16.2.11 failing on OSDs
by Lo Re Giuseppe 30 Mar '23

30 Mar '23

To add, on this, the issue seemed related to a process (ceph-volume) which was doing check operations on all devices. The systemctl osd service was timing out because of that and the osd daemon was going into error state. We noticed that version 17.2.5 had a change related to ceph-volume, in particular https://tracker.ceph.com/issues/57627. We decided to skip 16.2.11 and jump to 17.2.5. This second attempt went well, so the issue is now solved. Note: the upgrade 16.2.7 -> 16.2.11 went smoothly in a TDS cluster with identical OS/software, but much smaller, 3 nodes with a couple of disk each, so the issue seems really to be about the number of devices and nodes. Regards, Giuseppe On 30.03.23, 16:56, "Lo Re Giuseppe" <giuseppe.lore(a)cscs.ch <mailto:giuseppe.lore@cscs.ch>> wrote: Dear all, On one of our clusters I started the upgrade process from 16.2.7 to 16.2.11. Mon and mgr and crash processes were done easily/quickly, then at the first attempt of upgrading a OSD container the upgrade process stopped because of the OSD process is not able to start after the upgrade. Does anyone have any hint on how to unblock the upgrade? Some details below: Regards, Giuseppe I started the upgrade process with the cephadm command: “”” [root@naret-monitor01 ~]# ceph orch upgrade start --ceph-version 16.2.11 Initiating upgrade to quay.io/ceph/ceph:v16.2.11 “”” After a short time: “”” [root@naret-monitor01 ~]# ceph orch upgrade status { "target_image": quay.io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add<mailto:quay.io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add>, "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mon", "mgr" ], "progress": "64/2039 daemons upgraded", "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.4 on host naret-osd01 failed.", "is_paused": true } “”” The ceph health command reports: “”” [root@naret-monitor01 ~]# ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 2654362/6721382840 objects degraded (0.039%), 14 pgs degraded, 14 pgs undersized; Upgrading daemon osd.4 on host naret-osd01 failed. [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.22 on naret-osd01 is in error state [WRN] OSD_DOWN: 1 osds down osd.4 (root=default,host=naret-osd01) is down [WRN] PG_DEGRADED: Degraded data redundancy: 2654362/6721382840 objects degraded (0.039%), 14 pgs degraded, 14 pgs undersized pg 28.88 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1373,1337,1508,852,2147483647,483] pg 28.528 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1063,793,2147483647,931,338,1777] pg 28.594 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1208,891,1651,364,2147483647,53] pg 28.8b4 is stuck undersized for 6m, current state active+undersized+degraded, last acting [521,1273,1238,138,1539,2147483647] pg 28.a90 is stuck undersized for 6m, current state active+undersized+degraded, last acting [237,1665,1836,2147483647,192,1410] pg 28.ad6 is stuck undersized for 6m, current state active+undersized+degraded, last acting [870,466,350,885,1601,2147483647] pg 28.b34 is stuck undersized for 6m, current state active+undersized+degraded, last acting [920,1596,2147483647,115,201,941] pg 28.c14 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1389,424,2147483647,268,1646,632] pg 28.dba is stuck undersized for 6m, current state active+undersized+degraded, last acting [1099,561,2147483647,1806,1874,1145] pg 28.ee2 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1621,1904,1044,2147483647,1545,722] pg 29.163 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1883,2147483647,1509,1697,1187,235] pg 29.1c1 is stuck undersized for 6m, current state active+undersized+degraded, last acting [122,1226,962,1254,1215,2147483647] pg 29.254 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1782,1839,1545,412,196,2147483647] pg 29.2a1 is stuck undersized for 6m, current state active+undersized+degraded, last acting [370,2147483647,575,1423,1755,446] [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.4 on host naret-osd01 failed. Upgrade daemon: osd.4: cephadm exited with an error code: 1, stderr:Redeploy daemon osd.4 ... Non-zero exit code 1 from systemctl start ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4> systemctl: stderr Job for ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice> failed because a timeout was exceeded. systemctl: stderr See "systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice>" and "journalctl -xe" for details. Traceback (most recent call last): File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 9248, in <module> main() File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 9236, in main r = ctx.func(ctx) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 1990, in _default_image return func(ctx) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 5041, in command_deploy ports=daemon_ports) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 2952, in deploy_daemon c, osd_fsid=osd_fsid, ports=ports) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 3197, in deploy_daemon_units call_throws(ctx, ['systemctl', 'start', unit_name]) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 1657, in call_throws raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') RuntimeError: Failed command: systemctl start ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4>: Job for ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice> failed because a timeout was exceeded. See "systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice>" and "journalctl -xe" for details. “”” On the OSD server we have: “”” [root@naret-osd01 ~]# uname -a Linux naret-osd01 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Wed Dec 14 16:00:01 EST 2022 x86_64 x86_64 x86_64 GNU/Linux [root@naret-osd01 ~]# podman -v podman version 4.2.0 [root@naret-osd01 ~]# ceph -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) [root@naret-osd01 ~]# cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.7 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.7" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL=https://www.redhat.com/ <https://www.redhat.com/> DOCUMENTATION_URL=https://access.redhat.com/documentation/red_hat_enterpris… <https://access.redhat.com/documentation/red_hat_enterprise_linux/8/> BUG_REPORT_URL=https://bugzilla.redhat.com/ <https://bugzilla.redhat.com/> REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.7 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.7" “”” Systemctl says: “”” systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice> … ● ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice> - Ceph osd.4 for 63334166-d991-11eb-99de-40a6b72108d0 Loaded: loaded (/etc/systemd/system/ceph-63334166-d991-11eb-99de-40a6b72108d0@.service<mailto:/etc/systemd/system/ceph-63334166-d991-11eb-99de-40a6b72108d0@.service>; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Mon 2023-03-27 15:34:29 CEST; 6min ago Process: 730621 ExecStopPost=/bin/rm -f /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-pid> /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-cid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-cid> (code=exited, status=0/SUCCESS) Process: 730209 ExecStopPost=/bin/bash /var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/osd.4/unit.poststop (code=exited, status=0/SUCCESS) Process: 710355 ExecStartPre=/bin/rm -f /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-pid> /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-cid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-cid> (code=exited, status=0/SUCCESS) Main PID: 23025 (code=exited, status=0/SUCCESS) Tasks: 62 (limit: 1647878) Memory: 961.8M CGroup: /system.slice/system-ceph\x2d63334166\x2dd991\x2d11eb\x2d99de\x2d40a6b72108d0.slice/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice ├─libpod-payload-b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e │ └─754976 /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug └─runtime └─754965 /usr/bin/conmon --api-version 1 -c b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e -u b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata -p /run/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata/pidfile -n ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4 --exit-dir /run/libpod/exits --full-attach -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata/oci-log --conmon-pidfile /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice-pid> --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e6ae1700 1 osd.4 pg_epoch: 821628 pg[28.dbas2( v 821618'4657799 (819107'4647770,821618'4657799] local-lis/les=749842/749843 n=239290 ec=130297/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=7.751406670s) [1099,561,4,1806,1874,1145]p1099(0) r=2 lpr=821628 pi=[749842,821628)/1 crt=821618'4657799 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.039081573s@ mbc={} ps=[4~6]] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e8ae5700 1 osd.4 pg_epoch: 821628 pg[29.163s1( v 821572'139334 (776804'129273,821572'139334] local-lis/les=749851/749852 n=65683 ec=130801/130801 lis/c=821623/749851 les/c/f=821624/749852/0 sis=821628 pruub=8.023463249s) [1883,4,1509,1697,1187,235]p1883(0) r=1 lpr=821628 pi=[749851,821628)/1 crt=821572'139334 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.311203003s@ mbc={}] start_peering_interval up [1883,4,1509,1697,1187,235] -> [1883,4,1509,1697,1187,235], acting [1883,2147483647,1509,1697,1187,235] -> [1883,4,1509,1697,1187,235], acting_primary 1883(0) -> 1883, up_primary 1883(0) -> 1883, role -1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[29.2a1s1( v 821500'140649 (776804'130601,821500'140649] local-lis/les=749849/749850 n=65848 ec=130801/130801 lis/c=821623/749849 les/c/f=821624/749850/0 sis=821628 pruub=7.845988274s) [370,4,575,1423,1755,446]p370(0) r=1 lpr=821628 pi=[749849,821628)/1 crt=821500'140649 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.133728981s@ mbc={}] start_peering_interval up [370,4,575,1423,1755,446] -> [370,4,575,1423,1755,446], acting [370,2147483647,575,1423,1755,446] -> [370,4,575,1423,1755,446], acting_primary 370(0) -> 370, up_primary 370(0) -> 370, role -1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e8ae5700 1 osd.4 pg_epoch: 821628 pg[29.163s1( v 821572'139334 (776804'129273,821572'139334] local-lis/les=749851/749852 n=65683 ec=130801/130801 lis/c=821623/749851 les/c/f=821624/749852/0 sis=821628 pruub=8.023443222s) [1883,4,1509,1697,1187,235]p1883(0) r=1 lpr=821628 pi=[749851,821628)/1 crt=821572'139334 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.311203003s@ mbc={}] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[29.2a1s1( v 821500'140649 (776804'130601,821500'140649] local-lis/les=749849/749850 n=65848 ec=130801/130801 lis/c=821623/749849 les/c/f=821624/749850/0 sis=821628 pruub=7.845966339s) [370,4,575,1423,1755,446]p370(0) r=1 lpr=821628 pi=[749849,821628)/1 crt=821500'140649 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.133728981s@ mbc={}] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[28.8b4s5( v 821618'2906095 (817032'2896088,821618'2906095] local-lis/les=749842/749843 n=239377 ec=130295/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=8.158309937s) [521,1273,1238,138,1539,4]p521(0) r=5 lpr=821628 pi=[749842,821628)/1 crt=821618'2906095 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.446221352s@ mbc={} ps=[4~6]] start_peering_interval up [521,1273,1238,138,1539,4] -> [521,1273,1238,138,1539,4], acting [521,1273,1238,138,1539,2147483647] -> [521,1273,1238,138,1539,4], acting_primary 521(0) -> 521, up_primary 521(0) -> 521, role -1 -> 5, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[28.8b4s5( v 821618'2906095 (817032'2896088,821618'2906095] local-lis/les=749842/749843 n=239377 ec=130295/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=8.158291817s) [521,1273,1238,138,1539,4]p521(0) r=5 lpr=821628 pi=[749842,821628)/1 crt=821618'2906095 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.446221352s@ mbc={} ps=[4~6]] state<Start>: transitioning to Stray Mar 27 15:39:36 naret-osd01 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice>: Start request repeated too quickly. Mar 27 15:39:36 naret-osd01 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv <mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.serv>ice>: Failed with result 'timeout'. Mar 27 15:39:36 naret-osd01 systemd[1]: Failed to start Ceph osd.4 for 63334166-d991-11eb-99de-40a6b72108d0. “”” _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 0

Re: cephadm automatic sizing of WAL/DB on SSD
by Calhoun, Patrick 29 Mar '23

29 Mar '23

I think that the backported fix for this issue made it into ceph v16.2.11. https://ceph.io/en/news/blog/2023/v16-2-11-pacific-released/ "ceph-volume: Pacific backports (pr#47413, Guillaume Abrioux, Zack Cerza, Arthur Outhenin-Chalandre)" https://github.com/ceph/ceph/pull/47413/commits/4252cc44211f0ccebf388374744… -Patrick

1 0

ceph osd new: possible inconsistency whether UUID is a mandatory argument
by Oliver Schmidt 29 Mar '23

29 Mar '23

Hi everyone, I discovered a documentation inconsistency in Ceph Nautilus and would like to know whether this is still the case in the latest ceph release before reporting a bug. Unfortunately, I only have access to a Nautilus cluster right now. The quincy docs state [1]: > Create the OSD. If no UUID is given, it will be set automatically when the OSD starts up. The following command will output the OSD number, which you will need for subsequent steps: > >ceph osd create [{uuid} [{id}]] But the man pages [2] state that `ceph osd create` is deprecated in favour of `ceph osd new {<uuid>} {<id>} -i {<params.json>}`, with both uuid and id still being marked as optional parameters. But when actually running `ceph osd new` without a specified UUID, I get ``` Invalid command: missing required parameter uuid(<uuid>) osd new <uuid> {<osdname (id|osd.id)>} : Create a new OSD. If supplied, the `id` to be replaced needs to exist and have been previously destroyed. Reads secrets from JSON file via `-i <file>` (see man page). Error EINVAL: invalid command ``` under Nautilus. Is this still the case under Quincy, can someone reproduce this for me? Best regards Oliver Schmidt [1] https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/ [2] https://docs.ceph.com/en/quincy/man/8/ceph/#osd -- Oliver Schmidt · os(a)flyingcircus.io · Systems Engineer Flying Circus Internet Operations GmbH · http://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

1 0

5 host setup with NVMe's and HDDs
by Tino Todino 29 Mar '23

29 Mar '23

Hi folks. Just looking for some up to date advice please from the collective on how best to set up CEPH on 5 Proxmox hosts each configured with the following: AMD Ryzen 7 5800X CPU 64GB RAM 2x SSD (as ZFS boot disk for Proxmox) 1x 500GB NVMe for DB/WAL 1x 1TB NVMe as an OSD 1x 16TB SATA HDD as an OSD 2x 10GB NIC (One for Public and one for Cluster networks) 1 GB NIC for management interface The CEPH solution will be used primarily for storage of another Proxmox cluster's virtual machines and their data. We'd like a fast pool using the NVMe's for critical VMs and a slower HDD based pool for VM's that don't require such fast disk access and perhaps require more storage capacity. To expand in the future we will probably add more hosts in the same sort of configuration and/or replace NVMe/HDDs OSDs with more capacious ones. Ideas for configuration welcome please. Many thanks Tino Coastsense Ltd This E-mail is intended solely for the person or organisation to which it is addressed. It may contain privileged or confidential information and, if you are not the intended recipient, you must not copy, distribute or take any action in reliance upon it. Any views or opinions presented are solely those of the author and do not necessarily represent those of Marlan Maritime Technologies Ltd. If you have received this E-mail in error, please notify us as soon as possible and delete it from your computer. Marlan Maritime Technologies Ltd Registered in England & Wales 323 Mariners House, Norfolk Street, Liverpool. L1 0BG Company No. 08492427.

2 1

orphan multipart objects in Ceph cluster
by Ramin Najjarbashi 29 Mar '23

29 Mar '23

I hope this email finds you well. I wanted to share a recent experience I had with our Ceph cluster and get your feedback on a solution I came up with. Recently, we had some orphan objects stuck in our cluster that were not visible by any client like s3cmd, boto3, and mc. This caused some confusion for our users, as the sum of all objects in their buckets was much less than what we showed in the panel. We made some adjustments for them, but the issue persisted. As we have billions of objects in our cluster, using normal tools to find orphans was impossible. So, I came up with a tricky way to handle the situation. I created a bash script that identifies and removes the orphan objects using radosgw-admin and rados commands. Here is the script: https://gist.github.com/RaminNietzsche/b9baa06b69fc5f56d907f3c953769182 I am hoping to get some feedback from the community on this solution. Have any of you faced similar challenges with orphan objects in your Ceph clusters? Do you have any suggestions or improvements for my script? Thank you for your time and help.

2 1

osd_mclock_max_capacity_iops_ssd && multiple osd by nvme ?
by DERUMIER, Alexandre 29 Mar '23

29 Mar '23

Hi, I would like to advise to correctly tune osd_mclock_max_capacity_iops_ssd when you have multiple osd by nvme ? Does it need simply to divide the total iops of the nvme by number of osd ? But maybe it'll impact performance if more read/write are done on one of the osd ? I really don't known if we still need multiple osd by nvme in quincy ? Maybe it could be easier with a simple osd ? Regards, Alexandre

1 0

RGW can't create bucket
by kamil.madac＠gmail.com 29 Mar '23

29 Mar '23

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source

1 0

Adding new server to existing ceph cluster - with separate block.db on NVME
by Robert W. Eckert 29 Mar '23

29 Mar '23

Hi, I am trying to add a new server to an existing cluster, but cannot get the OSDs to create correctly When I try Cephadm ceph-volume lvm create, it returns nothing but the container info. [root@hiho ~]# cephadm ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Inferring fsid fe3a7cb0-69ca-11eb-8d45-c86000d08867 Using ceph image with id 'cc65afd6173a' and tag '<none>' created on 2022-10-17 23:41:41 +0000 UTC quay.io/ceph/ceph@sha256:2b73ccc9816e0a1ee1dfbe21ba9a8cc085210f1220f597b5050ebfcac4bdd346<mailto:quay.io/ceph/ceph@sha256:2b73ccc9816e0a1ee1dfbe21ba9a8cc085210f1220f597b5050ebfcac4bdd346> so I tried cephadm shell, and ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 41dafd4d-0579-4119-acca-6db31586a10f stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d0060d70) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-03-28T03:32:27.436+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d0063da0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-03-28T03:32:27.437+0000 7fa5d6253700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory stderr: 2023-03-28T03:32:27.437+0000 7fa5d6253700 -1 AuthRegistry(0x7fa5d6251ea0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx stderr: 2023-03-28T03:32:27.451+0000 7fa5ceffd700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-03-28T03:32:27.453+0000 7fa5cf7fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-03-28T03:32:27.473+0000 7fa5cffff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] stderr: 2023-03-28T03:32:27.474+0000 7fa5d6253700 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication stderr: [errno 13] RADOS permission denied (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id I then copy the key ring file into the container using scp, but by that time the orchestrator created OSDs on the drives, so I have to delete the OSDs and start over. Then if I get the timing just right, I get this (from within cephadm shell): [ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e6e316d4-670d-4a9b-a50c-bc14d57394a3 Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-4d95584a-df28-4e21-9480-09a13f1fb804 /dev/sdd stdout: Physical volume "/dev/sdd" successfully created. stdout: Volume group "ceph-4d95584a-df28-4e21-9480-09a13f1fb804" successfully created Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 953861 -n osd-block-e6e316d4-670d-4a9b-a50c-bc14d57394a3 ceph-4d95584a-df28-4e21-9480-09a13f1fb804 stdout: Logical volume "osd-block-e6e316d4-670d-4a9b-a50c-bc14d57394a3" created. Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 119209 -n osd-db-9fc4f199-2c95-4ca7-a35c-ef4b08c86804 ceph-948a633c-420e-4f55-8515-b33e1c0ef18c stderr: Volume group "ceph-948a633c-420e-4f55-8515-b33e1c0ef18c" has insufficient free space (0 extents): 119209 required. --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.12 --yes-i-really-mean-it stderr: purged osd.12 --> RuntimeError: Unable to find any LV for zapping OSD: 12 [ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 --> RuntimeError: Device /dev/sdd has a filesystem. [ceph: root@hiho bootstrap-osd]# ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/nvme0n1p3 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b93e1a8a-af88-431c-b705-f49d717b050f Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 /dev/sdd stdout: Physical volume "/dev/sdd" successfully created. stdout: Volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" successfully created Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 953861 -n osd-block-b93e1a8a-af88-431c-b705-f49d717b050f ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 stdout: Logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f" created. Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 /dev/nvme0n1p3 stdout: Physical volume "/dev/nvme0n1p3" successfully created. stdout: Volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" successfully created Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 119209 -n osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 stdout: Wiping ceph_bluestore signature on /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13. stdout: Logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13" created. Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5 Running command: /usr/bin/ln -s /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f /var/lib/ceph/osd/ceph-12/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-12/activate.monmap stderr: got monmap epoch 33 --> Creating keyring file for osd.12 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12/keyring Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12/ Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8 Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 12 --monmap /var/lib/ceph/osd/ceph-12/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 --osd-data /var/lib/ceph/osd/ceph-12/ --osd-uuid b93e1a8a-af88-431c-b705-f49d717b050f --setuser ceph --setgroup ceph stderr: 2023-03-28T03:23:21.180+0000 7f0a40fd63c0 -1 bluestore(/var/lib/ceph/osd/ceph-12/) _read_fsid unparsable uuid --> ceph-volume lvm prepare successful for: /dev/sdd Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f --path /var/lib/ceph/osd/ceph-12 --no-mon-config Running command: /usr/bin/ln -snf /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f /var/lib/ceph/osd/ceph-12/block Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/ln -snf /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 /var/lib/ceph/osd/ceph-12/block.db Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8 Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-12/block.db Running command: /usr/bin/chown -R ceph:ceph /dev/dm-8 Running command: /usr/bin/systemctl enable ceph-volume@lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume(a)lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f.service<mailto:/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-12-b93e1a8a-af88-431c-b705-f49d717b050f.service> -> /usr/lib/systemd/system/ceph-volume@.service<mailto:/usr/lib/systemd/system/ceph-volume@.service>. Running command: /usr/bin/systemctl enable --runtime ceph-osd@12 stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph-osd(a)12.service<mailto:/run/systemd/system/ceph-osd.target.wants/ceph-osd@12.service> -> /usr/lib/systemd/system/ceph-osd@.service<mailto:/usr/lib/systemd/system/ceph-osd@.service>. Running command: /usr/bin/systemctl start ceph-osd@12 stderr: Failed to connect to bus: No such file or directory --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.12 --yes-i-really-mean-it stderr: purged osd.12 --> Zapping: /dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f --> Unmounting /var/lib/ceph/osd/ceph-12 Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-12 stderr: umount: /var/lib/ceph/osd/ceph-12 unmounted Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83/osd-block-b93e1a8a-af88-431c-b705-f49d717b050f bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0771422 s, 136 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83 stderr: Removing ceph--e91c86b4--613f--45b0--b9d1--3bb76ed10f83-osd--block--b93e1a8a--af88--431c--b705--f49d717b050f (253:5) stderr: Releasing logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f" Archiving volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" metadata (seqno 5). stdout: Logical volume "osd-block-b93e1a8a-af88-431c-b705-f49d717b050f" successfully removed. stderr: Removing physical volume "/dev/sdd" from volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" stdout: Volume group "ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" successfully removed stderr: Creating volume group backup "/etc/lvm/backup/ceph-e91c86b4-613f-45b0-b9d1-3bb76ed10f83" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdd stdout: Labels on physical volume "/dev/sdd" successfully wiped. --> Zapping: /dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130/osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13 bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131332 s, 798 MB/s --> Only 1 LV left in VG, will proceed to destroy volume group ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-4abdb6f8-c891-43f8-8135-dd8470f80130 stderr: Removing ceph--4abdb6f8--c891--43f8--8135--dd8470f80130-osd--db--5bab4d99--e2e5--48e7--aee6--d1a2103b9d13 (253:8) stderr: Releasing logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13" Archiving volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" metadata (seqno 5). stdout: Logical volume "osd-db-5bab4d99-e2e5-48e7-aee6-d1a2103b9d13" successfully removed. stderr: Removing physical volume "/dev/nvme0n1p3" from volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" stdout: Volume group "ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" successfully removed stderr: Creating volume group backup "/etc/lvm/backup/ceph-4abdb6f8-c891-43f8-8135-dd8470f80130" (seqno 6). Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme0n1p3 stdout: Labels on physical volume "/dev/nvme0n1p3" successfully wiped. --> Zapping successful for OSD: 12 --> RuntimeError: command returned non-zero exit status: 1 I have 3 other servers with the split data and block db that I was able to install just using cephadm, so I am not sure what is off on this one. Is there any way to either add the block.db to the already created OSDs or get around the missing bootstrap.osd/ceph or manually configure the disk and block db? I have also tried ceph orch apply osd --all-available-devices --unmanaged=true to stop ceph from trying to take the OSD, but it still does Thanks, Rob

2 3

s3-select introduction blog / Trino integration
by Gal Salomon 28 Mar '23

28 Mar '23

Hi https://ceph.io/en/news/blog/2022/s3select-intro/ Recently I published a blog on s3-select. the Blog discusses what it is, and why it is required. the last paragraph discusses the Trino(analytic SQL utility) and its integration with Ceph/s3select. that integration is still on-work, and it is quite promising. Trino does not just provides a comprehensive SQL but also provides scalable processing for the SQL statements. we will be glad to hear your ideas and comments. Gal.

1 0

Re: Ceph cluster out of balance after adding OSDs
by Robert Sander 28 Mar '23

28 Mar '23

On 27.03.23 16:34, Pat Vaughan wrote: > Yes, all the OSDs are using the SSD device class. Do you have multiple CRUSH rules by chance? Are all pools using the same CRUSH rule? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 3

Unexpected slow read for HDD cluster (good write speed)
by Rafael Weingartner 28 Mar '23

28 Mar '23

Hello guys! I would like to ask if somebody has already experienced a similar situation. We have a new cluster with 5 nodes with the following setup: - 128 GB of RAM - 2 cpus Intel(R) Intel Xeon Silver 4210R - 1 NVME of 2 TB for the rocks DB caching - 5 HDDs of 14TB - 1 NIC dual port of 25GiB in BOND mode. We are starting with a single dual port NIC (the bond has 50GiB in total), the design has been prepared so a new NIC can be added, and a new BOND can be created, where we intend to offload the cluster network. Therefore, logically speaking, we already configured different VLANs and networks for public and cluster traffic of Ceph. We are using Ubuntu 20.04 with Ceph Octopus. It is a standard deployment that we are used to. During our initial validations and evaluations of the cluster, we are reaching write speeds between 250-300MB/s, which would be the ballpark for this kind of setup for HDDs with the NVME as Rocks.db cache (in our experience). However, the issue is the reading process. While reading, we barely hit the mark of 100MB/s; we would expect at least something similar to the write speed. These tests are being performed in a pool with a replication factor of 3. We have already checked the disks, and they all seem to be reading just fine. The network does not seem to be the bottleneck either (checked with atop while reading/writing to the cluster). Have you guys ever encountered similar situations? Do you have any tips for us to proceed with the troubleshooting? We suspect that we are missing some small tuning detail, which is affecting the read performance only, but so far we could not pinpoint it. Any help would be much appreciated :)

4 4

Almalinux 9
by Sere Gerrit 28 Mar '23

28 Mar '23

Hello, Has anyone used almalinux 9 to install ceph. Have you encountered problems? Other tips on this installation are also welcome. Regards, Gerrit

4 3

Question about adding SSDs
by Kyriazis, George 27 Mar '23

27 Mar '23

Hello ceph community, We have a ceph cluster (Proxmox based) with is HDD-based. We’ve had some performance and “slow MDS” issues while doing VM/CT backups from the Proxmox cluster, especially when rebalancing is going on at the same time. My thought is that one of following is going to improve performance / response: 1. Add an M.2 drive for DB store on each node 2. Migrate the cephfs metadata pool to SSDs We have ~25 nodes with ~3 OSDs per node. (1) is a lot of work and will cost more. (2) seems more risky (to me) since the metadata pool would have to be migrated (potential loss in transit?) Which one of the 2 solutions above will give us more bang for the buck, or just plain better performance? I would hate to implement (1) to find out that another solution would’ve been better. Any other solutions that I haven’t thought of? Thank you! George

2 2

rbd cp vs. rbd clone + rbd flatten
by Tony Liu 27 Mar '23

27 Mar '23

Hi, I want 1) copy a snapshot to an image, 2) no need to copy snapshots, 3) no dependency after copy, 4) all same image format 2. In that case, is rbd cp the same as rbd clone + rbd flatten? I ran some tests, seems like it, but want to confirm, in case of missing anything. Also, seems cp is a bit faster and flatten, is that true? Thanks! Tony

2 2

ceph orch ps shows version, container and image id as unknown
by Adiga, Anantha 27 Mar '23

27 Mar '23

Hi, Has anybody noticed this? ceph orch ps shows version, container and image id as unknown only for mon, mgr and osds. Ceph health is OK and all daemons are running fine. cephadm ls shows values for version, container and image id. root@cr21meg16ba0101:~# cephadm shell ceph orch ps Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID crash.cr21meg16ba0101 cr21meg16ba0101 running (4d) 2m ago 5w 7107k - 16.2.5 6e73176320aa 1001776a7c02 crash.cr21meg16ba0102 cr21meg16ba0102 running (5w) 2m ago 5w 63.8M - 16.2.5 6e73176320aa ecfd19d15dbb crash.cr21meg16ba0103 cr21meg16ba0103 running (5w) 2m ago 5w 8131k - 16.2.5 6e73176320aa c508ad3979b0 grafana.cr21meg16ba0101 cr21meg16ba0101 *:3000 running (4d) 2m ago 4d 58.0M - 6.7.4 be4c69a1aae8 ce2741a091c7 grafana.cr21meg16ba0102 cr21meg16ba0102 *:3000 running (4d) 2m ago 7d 53.0M - 6.7.4 be4c69a1aae8 c09f53b31999 grafana.cr21meg16ba0103 cr21meg16ba0103 *:3000 running (4d) 2m ago 7d 54.5M - 6.7.4 be4c69a1aae8 e58f6d9f44a2 haproxy.nfs.nfs-1.cr21meg16ba0101.cwsweq cr21meg16ba0101 *:2049,9049 running (4d) 2m ago 5w 66.7M - 2.3.21-3ce4ee0 7ecd3fda00f4 c5f4d94b5354 haproxy.nfs.nfs-1.cr21meg16ba0102.yodyxa cr21meg16ba0102 *:2049,9049 running (5w) 2m ago 5w 75.9M - 2.3.21-3ce4ee0 7ecd3fda00f4 0a6629e27463 haproxy.rgw.default.default.cr21meg16ba0101.ecpnxq cr21meg16ba0101 *:80,9050 running (4d) 2m ago 5w 102M - 2.3.21-3ce4ee0 7ecd3fda00f4 3c61d34b8b7d haproxy.rgw.default.default.cr21meg16ba0102.nffdzb cr21meg16ba0102 *:80,9050 running (5w) 2m ago 5w 114M - 2.3.21-3ce4ee0 7ecd3fda00f4 406ee603a311 haproxy.rgw.default.default.cr21meg16ba0103.lvypmb cr21meg16ba0103 *:80,9050 running (5w) 2m ago 5w 108M - 2.3.21-3ce4ee0 7ecd3fda00f4 a514c26c0a8e keepalived.nfs.nfs-1.cr21meg16ba0101.qpvesr cr21meg16ba0101 running (4d) 2m ago 5w 26.9M - 2.0.5 073e0c3cd1b9 c4003cb45da6 keepalived.nfs.nfs-1.cr21meg16ba0102.hedpuo cr21meg16ba0102 running (5w) 2m ago 5w 38.0M - 2.0.5 073e0c3cd1b9 b654e661493b keepalived.rgw.default.default.cr21meg16ba0101.biaqvq cr21meg16ba0101 running (4d) 2m ago 5w 39.2M - 2.0.5 073e0c3cd1b9 020c7cb700c4 keepalived.rgw.default.default.cr21meg16ba0102.dufodx cr21meg16ba0102 running (5w) 2m ago 5w 46.0M - 2.0.5 073e0c3cd1b9 fe218ecaf398 keepalived.rgw.default.default.cr21meg16ba0103.utplxz cr21meg16ba0103 running (5w) 2m ago 5w 45.1M - 2.0.5 073e0c3cd1b9 18a99c36ef29 mds.cephfs.cr21meg16ba0101.tmfknc cr21meg16ba0101 running (4d) 2m ago 5w 29.5M - 16.2.5 6e73176320aa e753a2498ccf mds.cephfs.cr21meg16ba0102.vdrcvi cr21meg16ba0102 running (5w) 2m ago 5w 212M - 16.2.5 6e73176320aa 925f151da4de mds.cephfs.cr21meg16ba0103.yacxeu cr21meg16ba0103 running (5w) 2m ago 5w 38.2M - 16.2.5 6e73176320aa 79599f7ca3c8 mgr.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - - <unknown> <unknown> <unknown> mon.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> nfs.nfs-1.0.63.cr21meg16ba0102.kkxpfh cr21meg16ba0102 *:12049 running (5w) 2m ago 5w 132M - 3.5 6e73176320aa fd617b99d70a nfs.nfs-1.1.0.cr21meg16ba0101.qhdmrq cr21meg16ba0101 *:12049 running (4d) 2m ago 5w 117M - 3.5 6e73176320aa 15e21064041d nfs.nfs-1.2.0.cr21meg16ba0103.wvmquo cr21meg16ba0103 *:12049 running (5w) 2m ago 5w 136M - 3.5 6e73176320aa edb9df63489d node-exporter.cr21meg16ba0101 cr21meg16ba0101 *:9100 running (4d) 2m ago 5w 79.3M - 0.17.0 b3e7f67a1480 e94d14814946 node-exporter.cr21meg16ba0102 cr21meg16ba0102 *:9100 running (5w) 2m ago 5w 102M - 0.17.0 b3e7f67a1480 86165fa7b916 node-exporter.cr21meg16ba0103 cr21meg16ba0103 *:9100 running (5w) 2m ago 5w 88.1M - 0.17.0 b3e7f67a1480 898008baaa66 osd.0 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.1 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.10 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.11 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.12 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.13 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.14 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.15 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.16 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.17 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.18 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.19 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.2 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.20 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.21 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.22 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.23 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.24 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.25 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.26 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.27 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.28 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.29 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.3 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.30 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.31 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.32 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.33 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.34 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.35 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.36 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.37 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.38 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.39 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.4 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.40 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.41 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.42 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.43 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.44 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.45 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.46 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.47 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.48 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.49 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.5 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.50 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.51 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.52 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.53 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.54 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.55 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.56 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.57 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.58 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.59 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.6 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.60 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.61 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.62 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.63 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.64 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.65 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.66 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.67 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.68 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.69 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.7 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.70 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.71 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.8 cr21meg16ba0102 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.9 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> prometheus.cr21meg16ba0101 cr21meg16ba0101 *:9095 running (4d) 2m ago 4d 358M - 2.7.2 2a68492b2466 2aeb3c975175 prometheus.cr21meg16ba0102 cr21meg16ba0102 *:9095 running (5d) 2m ago 5d 362M - 2.7.2 2a68492b2466 b5669b2b4756 rgw.default.default.cr21meg16ba0101.ppytbp cr21meg16ba0101 *:8080 running (4d) 2m ago 5w 188M - 16.2.5 6e73176320aa c13967bccdfc rgw.default.default.cr21meg16ba0102.fnmtmo cr21meg16ba0102 *:8080 running (5w) 2m ago 5w 168M - 16.2.5 6e73176320aa ddaa73b49c08 rgw.default.default.cr21meg16ba0103.xhxprp cr21meg16ba0103 *:8080 running (5w) 2m ago 5w 298M - 16.2.5 6e73176320aa cff4d77b7283 root@cr21meg16ba0101:~# cephadm shell ceph -s Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 cluster: id: a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe health: HEALTH_OK services: mon: 3 daemons, quorum cr21meg16ba0101,cr21meg16ba0102,cr21meg16ba0103 (age 4d) mgr: cr21meg16ba0101(active, since 42h), standbys: cr21meg16ba0103, cr21meg16ba0102 mds: 1/1 daemons up, 2 standby osd: 72 osds: 72 up (since 4d), 72 in (since 5w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 16 pools, 577 pgs objects: 3.86M objects, 12 TiB usage: 18 TiB used, 108 TiB / 126 TiB avail pgs: 577 active+clean io: client: 255 B/s rd, 115 KiB/s wr, 0 op/s rd, 3 op/s wr root@cr21meg16ba0101:~# cephadm shell ceph health Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 HEALTH_OK root@cr21meg16ba0101:~#

1 0

ceph orch ps shows unknown in version, container and image id columns
by anantha.adiga＠intel.com 27 Mar '23

27 Mar '23

Hi Has anybody noticed this issue? For all mgr, mon and osd daemons orch ps shows version, container and image ids as unknown. Ceph health is ok and all daemons are running fine. cephadm ls shows correct details of version, container and image ids . What could be the issue? and how to resolve it? ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) cephadm shell ceph -s Inferring fsid a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe Using recent ceph image ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 cluster: id: a6f52598-e5cd-4a08-8422-7b6fdb1d5dbe health: HEALTH_OK services: mon: 3 daemons, quorum cr21meg16ba0101,cr21meg16ba0102,cr21meg16ba0103 (age 4d) mgr: cr21meg16ba0101(active, since 42h), standbys: cr21meg16ba0103, cr21meg16ba0102 mds: 1/1 daemons up, 2 standby osd: 72 osds: 72 up (since 4d), 72 in (since 5w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 16 pools, 577 pgs objects: 3.86M objects, 12 TiB usage: 18 TiB used, 108 TiB / 126 TiB avail pgs: 577 active+clean io: client: 255 B/s rd, 115 KiB/s wr, 0 op/s rd, 3 op/s wr mds.cephfs.cr21meg16ba0103.yacxeu cr21meg16ba0103 running (5w) 2m ago 5w 38.2M - 16.2.5 6e73176320aa 79599f7ca3c8 mgr.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - - <unknown> <unknown> <unknown> mgr.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - - <unknown> <unknown> <unknown> mon.cr21meg16ba0101 cr21meg16ba0101 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0102 cr21meg16ba0102 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> mon.cr21meg16ba0103 cr21meg16ba0103 running 2m ago 5w - 2048M <unknown> <unknown> <unknown> nfs.nfs-1.0.63.cr21meg16ba0102.kkxpfh cr21meg16ba0102 *:12049 running (5w) 2m ago 5w 132M - 3.5 6e73176320aa fd617b99d70a nfs.nfs-1.1.0.cr21meg16ba0101.qhdmrq cr21meg16ba0101 *:12049 running (4d) 2m ago 5w 117M - 3.5 6e73176320aa 15e21064041d nfs.nfs-1.2.0.cr21meg16ba0103.wvmquo cr21meg16ba0103 *:12049 running (5w) 2m ago 5w 136M - 3.5 6e73176320aa edb9df63489d node-exporter.cr21meg16ba0101 cr21meg16ba0101 *:9100 running (4d) 2m ago 5w 79.3M - 0.17.0 b3e7f67a1480 e94d14814946 node-exporter.cr21meg16ba0102 cr21meg16ba0102 *:9100 running (5w) 2m ago 5w 102M - 0.17.0 b3e7f67a1480 86165fa7b916 node-exporter.cr21meg16ba0103 cr21meg16ba0103 *:9100 running (5w) 2m ago 5w 88.1M - 0.17.0 b3e7f67a1480 898008baaa66 osd.0 cr21meg16ba0101 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown> osd.1 cr21meg16ba0103 running 2m ago 5w - 22.0G <unknown> <unknown> <unknown>

1 0

Ceph cluster out of balance after adding OSDs
by Pat Vaughan 27 Mar '23

27 Mar '23

We setup a small Ceph cluster about 6 months ago with just 6x 200GB OSDs with one EC 4x2 pool. When we created that pool, we enabled pg_autoscale. The OSDs stayed pretty well balanced. After our developers released a new "feature" that caused the storage to balloon up to over 80%, we added another 6x 200GB OSDs. When we did that, we looked at the number of PGs for that pool, and found that there was only 1 for the rgw.data and rgw.log pools, and "osd pool autoscale-status" doesn't return anything, so it looks like that hasn't been working. The rebalance operation was extremely slow, and wasn't balancing out osd.0, so we bumped up the PGs for the rgw.data pool to 16. All the OSDs except osd.0 balanced out quickly, but that one OSDs utilization keeps climbing, and the number of misplaced objects is increasing, rather than decreasing. We set noscrub and nodeep-scrub so scrubbing wouldn't slow down the process. At this point, I don't want to do any more tuning to this cluster until we can get it back to a healthy state, but it's not fixing itself. I'm open to any ideas. Here's the output of ceph -s: cluster: id: 159d23e4-2a36-11ed-8b6e-fd27d573fa65 health: HEALTH_WARN 1 pools have many more objects per pg than average noscrub,nodeep-scrub flag(s) set 1 backfillfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 12 pgs backfill_toofull 7 pool(s) backfillfull services: mon: 3 daemons, quorum ceph3,ceph5,ceph6 (age 6h) mgr: ceph5.ksxevx(active, since 23h), standbys: ceph4.frkyyl, ceph6.slvpzl osd: 12 osds: 12 up (since 11h), 12 in (since 11h); 12 remapped pgs flags noscrub,nodeep-scrub rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 7 pools, 161 pgs objects: 28.61M objects, 211 GiB usage: 1.5 TiB used, 834 GiB / 2.3 TiB avail pgs: 91779228/171665865 objects misplaced (53.464%) 149 active+clean 12 active+remapped+backfill_toofull io: client: 11 KiB/s rd, 61 KiB/s wr, 11 op/s rd, 27 op/s wr progress: Global Recovery Event (23h) [=========================...] (remaining: 115m) ceph df: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 2.3 TiB 834 GiB 1.5 TiB 1.5 TiB 65.24 TOTAL 2.3 TiB 834 GiB 1.5 TiB 1.5 TiB 65.24 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 897 KiB 2 2.6 MiB 0.18 479 MiB .rgw.root 2 32 7.1 KiB 18 204 KiB 0.01 479 MiB charlotte.rgw.log 3 32 27 KiB 347 2.0 MiB 0.14 479 MiB charlotte.rgw.control 4 32 0 B 9 0 B 0 479 MiB charlotte.rgw.meta 5 32 9.7 KiB 16 167 KiB 0.01 479 MiB charlotte.rgw.buckets.data 6 16 734 GiB 28.61M 1.1 TiB 99.87 958 MiB charlotte.rgw.buckets.index 7 16 16 GiB 691 47 GiB 97.12 479 MiB ceph osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2.34357 root default -3 0.39059 host ceph1 0 ssd 0.19530 osd.0 up 0.89999 1.00000 1 ssd 0.19530 osd.1 up 1.00000 1.00000 -5 0.39059 host ceph2 6 ssd 0.19530 osd.6 up 1.00000 1.00000 7 ssd 0.19530 osd.7 up 1.00000 1.00000 -7 0.39059 host ceph3 2 ssd 0.19530 osd.2 up 1.00000 1.00000 8 ssd 0.19530 osd.8 up 1.00000 1.00000 -9 0.39059 host ceph4 3 ssd 0.19530 osd.3 up 1.00000 1.00000 9 ssd 0.19530 osd.9 up 1.00000 1.00000 -11 0.39059 host ceph5 4 ssd 0.19530 osd.4 up 1.00000 1.00000 10 ssd 0.19530 osd.10 up 1.00000 1.00000 -13 0.39059 host ceph6 5 ssd 0.19530 osd.5 up 1.00000 1.00000 11 ssd 0.19530 osd.11 up 1.00000 1.00000 ceph osd df: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 ssd 0.19530 0.89999 200 GiB 190 GiB 130 GiB 12 GiB 48 GiB 10 GiB 94.94 1.46 52 up 1 ssd 0.19530 1.00000 200 GiB 7.3 GiB 9.8 MiB 6.4 GiB 858 MiB 193 GiB 3.64 0.06 42 up 6 ssd 0.19530 1.00000 200 GiB 148 GiB 97 GiB 14 GiB 38 GiB 52 GiB 74.06 1.14 51 up 7 ssd 0.19530 1.00000 200 GiB 133 GiB 97 GiB 2 KiB 35 GiB 67 GiB 66.47 1.02 43 up 2 ssd 0.19530 1.00000 200 GiB 134 GiB 97 GiB 12 KiB 37 GiB 66 GiB 66.94 1.03 40 up 8 ssd 0.19530 1.00000 200 GiB 136 GiB 97 GiB 2.2 GiB 36 GiB 64 GiB 67.85 1.04 40 up 3 ssd 0.19530 1.00000 200 GiB 134 GiB 97 GiB 4 KiB 37 GiB 66 GiB 66.95 1.03 41 up 9 ssd 0.19530 1.00000 200 GiB 138 GiB 97 GiB 5.2 GiB 36 GiB 62 GiB 69.19 1.06 49 up 4 ssd 0.19530 1.00000 200 GiB 137 GiB 97 GiB 4.3 GiB 36 GiB 63 GiB 68.62 1.05 42 up 10 ssd 0.19530 1.00000 200 GiB 139 GiB 97 GiB 5.5 GiB 36 GiB 61 GiB 69.31 1.06 48 up 5 ssd 0.19530 1.00000 200 GiB 134 GiB 97 GiB 7 KiB 38 GiB 66 GiB 67.13 1.03 34 up 11 ssd 0.19530 1.00000 200 GiB 136 GiB 97 GiB 2.2 GiB 36 GiB 64 GiB 67.80 1.04 49 up TOTAL 2.3 TiB 1.5 TiB 1.1 TiB 52 GiB 414 GiB 834 GiB 65.24 MIN/MAX VAR: 0.06/1.46 STDDEV: 19.95 Thanks in advance if anyone has any suggestions.

2 1

EC profiles where m>k (EC 8+12)
by Fabien Sirjean 27 Mar '23

27 Mar '23

Hi Ceph users! I've been proposed an interesting EC setup I hadn't thought about before. Scenario is : we have two server rooms and want to store ~4PiB with the ability to loose 1 server room without loss of data or RW availability. For the context, performance is not needed (cold storage mostly, used as a big filesystem). The idea is to use EC 8+12 over 24 servers (12 on each server room), so if we loose 1 room we still have half of the EC parts (10/20) and are able to loose 2 more servers before reaching the point where we loose data. I find this pretty elegant when working on a two-sites context, as efficiency is 40% (better than 33% three times replication) and the redundancy is good. What do you think of this setup ? Did you ever used EC profiles with M > K ? Thanks for sharing your thoughts! Cheers, Fabien

5 5

Workload performance varying between 2 executions
by Nguetchouang Ngongang Kevin 27 Mar '23

27 Mar '23

Good morning. I just setup a ceph environment, 9 storage nodes and i mount it on a cephfs on a 10th independent node. I execute a fio workload once i got a 3Mb/s throughput. When i reexecute the same workload after a certain time i got this time a 9Mb/s throughput. Do you know why this is happening ? Cordially, -- Nguetchouang Ngongang Kevin ENS de Lyon https://perso.ens-lyon.fr/kevin.nguetchouang/

1 0

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

Unbalanced OSDs when pg_autoscale enabled
by 郑亮 24 Mar '23

24 Mar '23

Hi all, I have a 9 node cluster running *Pacific 16.2.10*. OSDs live on 9 of the nodes with each one having 4 x 1.8T ssd and 8 x 10.9T hdd for a total of 108 OSDs. We create three crush roots as belows. 1. The hdds (8x9=72) of all nodes form a large crush root, which is used as a data pool, and object storage and cephfs share this crush root. 2. Take 3 ssds from the 4 ssds of each node as rbd block storage. 3. An ssd on each remaining node is used as an index pool for cephfs and object storage. [root@node01 smd]# ceph osd treeID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -92 15.71910 root root-1c31624a-ad18-445e-8e42-86b71c1fd76f -112 1.74657 host node01-fa2cdf3e-7212-4b5f-b62a-3ab1e803547f 13 ssd 1.74657 osd.13 up 1.00000 1.00000 -103 1.74657 host node02-4e232f27-fe4b-4d0e-bd2a-67d5006a0cdd 34 ssd 1.74657 osd.34 up 1.00000 1.00000 -109 1.74657 host node03-3ae63d7a-9f65-4bea-b2ba-ff3fe342753d 28 ssd 1.74657 osd.28 up 1.00000 1.00000 -118 1.74657 host node04-37a3f92a-f6d8-41f9-a774-3069fc2f50b8 54 ssd 1.74657 osd.54 up 1.00000 1.00000 -106 1.74657 host node05-f667fa27-cc13-4b93-ad56-5dc4c31ffd77 53 ssd 1.74657 osd.53 up 1.00000 1.00000 -91 1.74657 host node06-3808c8f6-8e10-47c7-8456-62c1e0e800ed 61 ssd 1.74657 osd.61 up 1.00000 1.00000 -97 1.74657 host node07-78216b0d-0999-44e8-905d-8737a5f6f51f 50 ssd 1.74657 osd.50 up 1.00000 1.00000 -115 1.74657 host node08-947bd556-fb06-497d-8f2c-c4a679d2b06f 86 ssd 1.74657 osd.86 up 1.00000 1.00000 -100 1.74657 host node09-d9ae9046-0716-454f-ba0c-b03cf9986ba8 85 ssd 1.74657 osd.85 up 1.00000 1.00000 -38 785.80701 root root-6041a4dc-7c9a-44ed-999c-a847cca81012 -85 87.31189 host node01-e5646053-2cf8-4ba5-90d5-bb1a63b1234c 1 hdd 10.91399 osd.1 up 1.00000 1.00000 22 hdd 10.91399 osd.22 up 0.90002 1.00000 31 hdd 10.91399 osd.31 up 1.00000 1.00000 51 hdd 10.91399 osd.51 up 1.00000 1.00000 60 hdd 10.91399 osd.60 up 1.00000 1.00000 70 hdd 10.91399 osd.70 up 1.00000 1.00000 78 hdd 10.91399 osd.78 up 1.00000 1.00000 96 hdd 10.91399 osd.96 up 1.00000 1.00000 -37 87.31189 host node02-be9925fd-60de-4147-81eb-720d7145715f 9 hdd 10.91399 osd.9 up 1.00000 1.00000 19 hdd 10.91399 osd.19 up 1.00000 1.00000 29 hdd 10.91399 osd.29 up 1.00000 1.00000 47 hdd 10.91399 osd.47 up 1.00000 1.00000 56 hdd 10.91399 osd.56 up 1.00000 1.00000 65 hdd 10.91399 osd.65 up 1.00000 1.00000 88 hdd 10.91399 osd.88 up 1.00000 1.00000 98 hdd 10.91399 osd.98 up 1.00000 1.00000 -52 87.31189 host node03-7828653d-6033-4e88-92b0-d8709b0ab218 2 hdd 10.91399 osd.2 up 1.00000 1.00000 30 hdd 10.91399 osd.30 up 1.00000 1.00000 40 hdd 10.91399 osd.40 up 1.00000 1.00000 48 hdd 10.91399 osd.48 up 1.00000 1.00000 58 hdd 10.91399 osd.58 up 1.00000 1.00000 74 hdd 10.91399 osd.74 up 1.00000 1.00000 83 hdd 10.91399 osd.83 up 1.00000 1.00000 92 hdd 10.91399 osd.92 up 1.00000 1.00000 -46 87.31189 host node04-e986c3fc-a21b-44ff-9b02-b60b82ee63d7 12 hdd 10.91399 osd.12 up 1.00000 1.00000 23 hdd 10.91399 osd.23 up 1.00000 1.00000 32 hdd 10.91399 osd.32 up 1.00000 1.00000 43 hdd 10.91399 osd.43 up 1.00000 1.00000 52 hdd 10.91399 osd.52 up 1.00000 1.00000 71 hdd 10.91399 osd.71 up 1.00000 1.00000 95 hdd 10.91399 osd.95 up 1.00000 1.00000 104 hdd 10.91399 osd.104 up 1.00000 1.00000 -88 87.31189 host node05-fe31d85f-b3b9-4393-b24b-030dbcdfacea 3 hdd 10.91399 osd.3 up 1.00000 1.00000 24 hdd 10.91399 osd.24 up 1.00000 1.00000 33 hdd 10.91399 osd.33 up 1.00000 1.00000 45 hdd 10.91399 osd.45 up 1.00000 1.00000 69 hdd 10.91399 osd.69 up 1.00000 1.00000 79 hdd 10.91399 osd.79 up 1.00000 1.00000 89 hdd 10.91399 osd.89 up 1.00000 1.00000 99 hdd 10.91399 osd.99 up 1.00000 1.00000 -55 87.31189 host node06-6f16ba4b-0082-472a-b243-b1a058070918 5 hdd 10.91399 osd.5 up 1.00000 1.00000 15 hdd 10.91399 osd.15 up 1.00000 1.00000 25 hdd 10.91399 osd.25 up 1.00000 1.00000 44 hdd 10.91399 osd.44 up 1.00000 1.00000 63 hdd 10.91399 osd.63 up 1.00000 1.00000 72 hdd 10.91399 osd.72 up 1.00000 1.00000 81 hdd 10.91399 osd.81 up 1.00000 1.00000 91 hdd 10.91399 osd.91 up 1.00000 1.00000 -43 87.31189 host node07-5dee846a-2814-4e04-bcfd-ff689d49795c 7 hdd 10.91399 osd.7 up 1.00000 1.00000 37 hdd 10.91399 osd.37 up 1.00000 1.00000 46 hdd 10.91399 osd.46 up 1.00000 1.00000 55 hdd 10.91399 osd.55 up 1.00000 1.00000 64 hdd 10.91399 osd.64 up 1.00000 1.00000 84 hdd 10.91399 osd.84 up 1.00000 1.00000 94 hdd 10.91399 osd.94 up 1.00000 1.00000 102 hdd 10.91399 osd.102 up 1.00000 1.00000 -58 87.31189 host node08-2d6b7ab3-2067-4e94-b77c-24d6e626e396 6 hdd 10.91399 osd.6 up 1.00000 1.00000 18 hdd 10.91399 osd.18 up 1.00000 1.00000 41 hdd 10.91399 osd.41 up 1.00000 1.00000 49 hdd 10.91399 osd.49 up 1.00000 1.00000 59 hdd 10.91399 osd.59 up 1.00000 1.00000 68 hdd 10.91399 osd.68 up 1.00000 1.00000 97 hdd 10.91399 osd.97 up 1.00000 1.00000 105 hdd 10.91399 osd.105 up 1.00000 1.00000 -49 87.31189 host node09-e6bf0642-e3b4-48f0-9d0e-3d87ceacced8 8 hdd 10.91399 osd.8 up 1.00000 1.00000 17 hdd 10.91399 osd.17 up 1.00000 1.00000 27 hdd 10.91399 osd.27 up 1.00000 1.00000 38 hdd 10.91399 osd.38 up 1.00000 1.00000 57 hdd 10.91399 osd.57 up 1.00000 1.00000 67 hdd 10.91399 osd.67 up 1.00000 1.00000 93 hdd 10.91399 osd.93 up 1.00000 1.00000 103 hdd 10.91399 osd.103 up 1.00000 1.00000 -8 47.16115 root root-ea7a1878-722e-49d5-8a91-c618a6aefe29 -13 5.24013 host node01-4c465825-6bd1-42a5-b087-51a82cb2865c 0 ssd 1.74657 osd.0 up 0.95001 1.00000 11 ssd 1.74699 osd.11 up 1.00000 1.00000 16 ssd 1.74657 osd.16 up 1.00000 1.00000 -22 5.24013 host node02-0e3e418d-9129-44ee-8453-53680171270e 4 ssd 1.74657 osd.4 up 1.00000 1.00000 14 ssd 1.74657 osd.14 up 0.95001 1.00000 82 ssd 1.74699 osd.82 up 1.00000 1.00000 -28 5.24013 host node03-274397b4-8559-4a17-a9eb-1c16920ad432 10 ssd 1.74657 osd.10 up 1.00000 1.00000 76 ssd 1.74657 osd.76 up 1.00000 1.00000 100 ssd 1.74699 osd.100 up 0.95001 1.00000 -31 5.24013 host node04-df05c3cc-08ef-426e-85b8-088cb8c1b4e2 20 ssd 1.74657 osd.20 up 1.00000 1.00000 39 ssd 1.74657 osd.39 up 0.95001 1.00000 90 ssd 1.74699 osd.90 up 0.95001 1.00000 -25 5.24013 host node05-d07e7e92-8290-49ba-b09c-ad34bddb1eae 26 ssd 1.74657 osd.26 up 0.95001 1.00000 62 ssd 1.74699 osd.62 up 1.00000 1.00000 73 ssd 1.74657 osd.73 up 1.00000 1.00000 -16 5.24013 host node06-a7a40371-a128-4ab1-90a2-62c99e040036 42 ssd 1.74657 osd.42 up 0.95001 1.00000 75 ssd 1.74657 osd.75 up 0.95001 1.00000 107 ssd 1.74699 osd.107 up 1.00000 1.00000 -7 5.24013 host node07-971dee6e-dec4-4c0a-86d1-54d0b23832bd 21 ssd 1.74699 osd.21 up 1.00000 1.00000 66 ssd 1.74657 osd.66 up 0.95001 1.00000 77 ssd 1.74657 osd.77 up 1.00000 1.00000 -19 5.24013 host node08-9a47282b-9530-4cc7-9e29-7f6c0b2f5184 35 ssd 1.74699 osd.35 up 1.00000 1.00000 80 ssd 1.74657 osd.80 up 1.00000 1.00000 101 ssd 1.74657 osd.101 up 1.00000 1.00000 -34 5.24013 host node09-f09a8013-2426-4ced-b9c7-02c06ca9d6fc 36 ssd 1.74699 osd.36 up 0.95001 1.00000 87 ssd 1.74657 osd.87 up 1.00000 1.00000 106 ssd 1.74657 osd.106 up 1.00000 1.00000 After ceph has been running normally for a period of time, I found that the distribution of osd data is seriously uneven, and autoscale PGs were turned on in my envs. My hdds are identical, but utilization ranges from 6.14% to 74.35% root@node09:/# ceph osd df | grep hdd ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.2 TiB 15 KiB 53 GiB 5.7 TiB 48.15 1.38 174 up 22 hdd 10.91399 0.90002 11 TiB 8.1 TiB 8.0 TiB 12 KiB 71 GiB 2.8 TiB 74.35 2.13 160 up 31 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 13 KiB 27 GiB 8.0 TiB 27.14 0.78 179 up 51 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 9 KiB 36 GiB 7.2 TiB 34.12 0.98 173 up 60 hdd 10.91399 1.00000 11 TiB 973 GiB 962 GiB 14 KiB 11 GiB 10 TiB 8.71 0.25 167 up 70 hdd 10.91399 1.00000 11 TiB 2.0 TiB 2.0 TiB 11 KiB 3.9 GiB 8.9 TiB 18.42 0.53 173 up 78 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 12 KiB 28 GiB 7.9 TiB 27.18 0.78 173 up 96 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 10 KiB 36 GiB 7.2 TiB 34.15 0.98 176 up 9 hdd 10.91399 1.00000 11 TiB 2.8 TiB 2.8 TiB 16 KiB 13 GiB 8.1 TiB 25.38 0.73 178 up 19 hdd 10.91399 1.00000 11 TiB 3.8 TiB 3.8 TiB 13 KiB 43 GiB 7.1 TiB 35.04 1.00 177 up 29 hdd 10.91399 1.00000 11 TiB 5.1 TiB 5.0 TiB 14 KiB 38 GiB 5.9 TiB 46.38 1.33 182 up 47 hdd 10.91399 1.00000 11 TiB 686 GiB 684 GiB 9 KiB 1.4 GiB 10 TiB 6.14 0.18 172 up 56 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.16 0.98 174 up 65 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 7 KiB 54 GiB 5.0 TiB 54.27 1.55 183 up 88 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 12 KiB 44 GiB 6.4 TiB 41.17 1.18 173 up 98 hdd 10.91399 1.00000 11 TiB 7.3 TiB 7.2 TiB 13 KiB 56 GiB 3.7 TiB 66.49 1.90 179 up 2 hdd 10.91399 1.00000 11 TiB 1.5 TiB 1.5 TiB 12 KiB 18 GiB 9.4 TiB 14.02 0.40 177 up 30 hdd 10.91399 1.00000 11 TiB 4.4 TiB 4.4 TiB 10 KiB 37 GiB 6.5 TiB 40.29 1.15 180 up 40 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 10 KiB 11 GiB 9.5 TiB 13.14 0.38 172 up 48 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 12 KiB 11 GiB 9.5 TiB 13.17 0.38 168 up 58 hdd 10.91399 1.00000 11 TiB 7.5 TiB 7.4 TiB 14 KiB 70 GiB 3.5 TiB 68.31 1.96 182 up 74 hdd 10.91399 1.00000 11 TiB 6.0 TiB 6.0 TiB 10 KiB 61 GiB 4.9 TiB 55.15 1.58 181 up 83 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 22 KiB 19 GiB 8.7 TiB 20.13 0.58 174 up 92 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 11 KiB 27 GiB 8.0 TiB 27.15 0.78 171 up 12 hdd 10.91399 1.00000 11 TiB 5.1 TiB 5.0 TiB 9 KiB 38 GiB 5.9 TiB 46.37 1.33 176 up 23 hdd 10.91399 1.00000 11 TiB 4.4 TiB 4.4 TiB 22 KiB 37 GiB 6.5 TiB 40.24 1.15 173 up 32 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 10 KiB 19 GiB 8.7 TiB 20.13 0.58 176 up 43 hdd 10.91399 1.00000 11 TiB 2.0 TiB 2.0 TiB 10 KiB 4.4 GiB 8.9 TiB 18.41 0.53 176 up 52 hdd 10.91399 1.00000 11 TiB 3.6 TiB 3.6 TiB 12 KiB 29 GiB 7.3 TiB 33.27 0.95 171 up 71 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13 0.78 170 up 95 hdd 10.91399 1.00000 11 TiB 5.4 TiB 5.3 TiB 10 KiB 60 GiB 5.6 TiB 49.04 1.40 173 up104 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 8 KiB 36 GiB 7.2 TiB 34.13 0.98 171 up 3 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 9 KiB 19 GiB 8.7 TiB 20.14 0.58 173 up 24 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 9 KiB 19 GiB 8.7 TiB 20.12 0.58 170 up 33 hdd 10.91399 1.00000 11 TiB 6.8 TiB 6.7 TiB 14 KiB 69 GiB 4.1 TiB 62.15 1.78 181 up 45 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.27 0.75 178 up 69 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 15 KiB 20 GiB 8.0 TiB 26.25 0.75 173 up 79 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 28 KiB 54 GiB 5.0 TiB 54.25 1.55 178 up 89 hdd 10.91399 1.00000 11 TiB 2.1 TiB 2.1 TiB 16 KiB 11 GiB 8.8 TiB 19.25 0.55 173 up 99 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 13 KiB 44 GiB 6.4 TiB 41.14 1.18 174 up 5 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 14 KiB 19 GiB 8.7 TiB 20.12 0.58 172 up 15 hdd 10.91399 1.00000 11 TiB 7.3 TiB 7.2 TiB 7 KiB 56 GiB 3.7 TiB 66.48 1.90 177 up 25 hdd 10.91399 1.00000 11 TiB 1.4 TiB 1.4 TiB 11 KiB 10 GiB 9.5 TiB 13.13 0.38 176 up 44 hdd 10.91399 1.00000 11 TiB 3.1 TiB 3.0 TiB 9 KiB 34 GiB 7.9 TiB 28.03 0.80 179 up 63 hdd 10.91399 1.00000 11 TiB 2.9 TiB 2.8 TiB 10 KiB 20 GiB 8.0 TiB 26.26 0.75 178 up 72 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 12 KiB 36 GiB 7.2 TiB 34.13 0.98 180 up 81 hdd 10.91399 1.00000 11 TiB 6.7 TiB 6.6 TiB 11 KiB 62 GiB 4.2 TiB 61.30 1.76 181 up 91 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 16 KiB 19 GiB 8.7 TiB 20.11 0.58 176 up 7 hdd 10.91399 1.00000 11 TiB 4.6 TiB 4.5 TiB 13 KiB 52 GiB 6.3 TiB 42.03 1.20 180 up 37 hdd 10.91399 1.00000 11 TiB 5.3 TiB 5.2 TiB 14 KiB 52 GiB 5.7 TiB 48.16 1.38 182 up 46 hdd 10.91399 1.00000 11 TiB 784 GiB 775 GiB 14 KiB 8.8 GiB 10 TiB 7.01 0.20 173 up 55 hdd 10.91399 1.00000 11 TiB 5.9 TiB 5.9 TiB 20 KiB 54 GiB 5.0 TiB 54.25 1.55 177 up 64 hdd 10.91399 1.00000 11 TiB 3.6 TiB 3.6 TiB 18 KiB 28 GiB 7.3 TiB 33.23 0.95 176 up 84 hdd 10.91399 1.00000 11 TiB 784 GiB 775 GiB 10 KiB 9.0 GiB 10 TiB 7.01 0.20 176 up 94 hdd 10.91399 1.00000 11 TiB 2.1 TiB 2.1 TiB 8 KiB 12 GiB 8.8 TiB 19.28 0.55 169 up102 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 10 KiB 28 GiB 8.0 TiB 27.13 0.78 170 up 6 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 23 KiB 44 GiB 6.4 TiB 41.14 1.18 181 up 18 hdd 10.91399 1.00000 11 TiB 6.0 TiB 6.0 TiB 9 KiB 61 GiB 4.9 TiB 55.15 1.58 183 up 41 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.12 0.78 179 up 49 hdd 10.91399 1.00000 11 TiB 785 GiB 775 GiB 19 KiB 9.5 GiB 10 TiB 7.02 0.20 176 up 59 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 8 KiB 19 GiB 8.7 TiB 20.12 0.58 178 up 68 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB 36 GiB 7.2 TiB 34.15 0.98 172 up 97 hdd 10.91399 1.00000 11 TiB 3.0 TiB 2.9 TiB 15 KiB 27 GiB 8.0 TiB 27.13 0.78 178 up105 hdd 10.91399 1.00000 11 TiB 3.7 TiB 3.7 TiB 13 KiB 36 GiB 7.2 TiB 34.14 0.98 179 up 8 hdd 10.91399 1.00000 11 TiB 6.6 TiB 6.5 TiB 13 KiB 55 GiB 4.3 TiB 60.40 1.73 181 up 17 hdd 10.91399 1.00000 11 TiB 4.6 TiB 4.5 TiB 14 KiB 52 GiB 6.3 TiB 42.03 1.20 177 up 27 hdd 10.91399 1.00000 11 TiB 783 GiB 774 GiB 8 KiB 9.4 GiB 10 TiB 7.01 0.20 176 up 38 hdd 10.91399 1.00000 11 TiB 2.2 TiB 2.2 TiB 15 KiB 19 GiB 8.7 TiB 20.11 0.58 174 up 57 hdd 10.91399 1.00000 11 TiB 2.3 TiB 2.3 TiB 12 KiB 26 GiB 8.6 TiB 21.01 0.60 177 up 67 hdd 10.91399 1.00000 11 TiB 4.5 TiB 4.4 TiB 9 KiB 44 GiB 6.4 TiB 41.14 1.18 176 up 93 hdd 10.91399 1.00000 11 TiB 3.1 TiB 3.0 TiB 7 KiB 35 GiB 7.9 TiB 28.01 0.80 171 up103 hdd 10.91399 1.00000 11 TiB 4.2 TiB 4.2 TiB 10 KiB 22 GiB 6.7 TiB 38.52 1.10 176 up three pools (deeproute-replica-hdd-pool, os-dsglczutvqsgowpz.rgw.buckets.data, and cephfs-replicated-pool ) are set to use the same device class hdd. I noticed that the *effective ratio* of the three pools is very different, not sure if it will affect the data rebalancing. [root@node01 smd]# ceph osd pool autoscale-statusPOOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK device_health_metrics 278.5M 3.0 48289G 0.0000 1.0 1 on False deeproute-replica-hdd-pool 12231M 3.0 785.8T 0.9901 50.0000 0.9901 1.0 4096 on False deeproute-replica-ssd-pool 12333G 3.0 48289G 0.9901 50.0000 0.9901 1.0 1024 on False .rgw.root 5831 3.0 48289G 0.0099 0.5000 0.0099 1.0 8 on False default.rgw.log 182 3.0 48289G 0.0000 1.0 32 on False default.rgw.control 0 3.0 48289G 0.0000 1.0 32 on False default.rgw.meta 0 3.0 48289G 0.0000 4.0 8 on False os-dsglczutvqsgowpz.rgw.control 0 3.0 16096G 0.1667 0.5000 0.1667 1.0 64 on False os-dsglczutvqsgowpz.rgw.meta 99.81k 3.0 16096G 0.1667 0.5000 0.1667 1.0 64 on False os-dsglczutvqsgowpz.rgw.buckets.index 59437M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.buckets.non-ec 498.7M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.log 1331M 3.0 16096G 0.1667 0.5000 0.1667 1.0 32 on False os-dsglczutvqsgowpz.rgw.buckets.data 143.1T 1.3333333730697632 785.8T 0.2429 0.5000 0.0099 1.0 32 on False cephfs-metadata 3161M 3.0 16096G 0.0006 4.0 32 on False cephfs-replicated-pool 21862G 3.0 785.8T 0.0815 1.0 32 on False .nfs 84599 3.0 48289G 0.0000 1.0 32 on False os-dsglczutvqsgowpz.rgw.otp 0 3.0 16096G 0.1667 0.5000 [root@node01 smd]# ceph df--- RAW STORAGE ---CLASS SIZE AVAIL USED RAW USED %RAW USEDhdd 786 TiB 545 TiB 240 TiB 240 TiB 30.60ssd 63 TiB 25 TiB 38 TiB 38 TiB 59.95TOTAL 849 TiB 571 TiB 278 TiB 278 TiB 32.77 --- POOLS ---POOL ID PGS STORED OBJECTS USED %USED MAX AVAILdevice_health_metrics 1 1 280 MiB 135 841 MiB 0.02 1.8 TiBdeeproute-replica-hdd-pool 11 4096 12 GiB 9.44k 36 GiB 0.02 69 TiBdeeproute-replica-ssd-pool 12 1024 12 TiB 3.43M 37 TiB 87.62 1.8 TiB.rgw.root 25 8 5.7 KiB 20 228 KiB 0 1.8 TiBdefault.rgw.log 26 32 182 B 2 24 KiB 0 1.8 TiBdefault.rgw.control 27 32 0 B 8 0 B 0 1.8 TiBdefault.rgw.meta 28 8 0 B 0 0 B 0 1.8 TiBos-dsglczutvqsgowpz.rgw.control 29 64 0 B 8 0 B 0 4.9 TiBos-dsglczutvqsgowpz.rgw.meta 30 64 100 KiB 456 4.3 MiB 0 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.index 31 32 58 GiB 10.40k 174 GiB 1.15 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.non-ec 32 32 500 MiB 247.78k 4.3 GiB 0.03 4.9 TiBos-dsglczutvqsgowpz.rgw.log 33 32 689 MiB 432 2.0 GiB 0.01 4.9 TiBos-dsglczutvqsgowpz.rgw.buckets.data 34 32 130 TiB 134.96M 174 TiB 45.69 155 TiBcephfs-metadata 35 32 3.1 GiB 1.67M 9.3 GiB 0.06 4.9 TiBcephfs-replicated-pool 36 32 21 TiB 8.25M 64 TiB 23.79 69 TiB.nfs 37 32 89 KiB 9 341 KiB 0 1.8 TiBos-dsglczutvqsgowpz.rgw.otp 38 8 0 B 0 0 B 0 4.9 TiB [root@node01 smd]# ceph -s cluster: id: f7f1c8ba-f793-436b-bb73-0964108a30c1 health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 7w) mgr: a(active, since 2w), standbys: b mds: 1/1 daemons up, 1 hot standby osd: 108 osds: 108 up (since 5h), 108 in (since 4M); 1 remapped pgs rgw: 9 daemons active (9 hosts, 1 zones) data: volumes: 1/1 healthy pools: 17 pools, 5561 pgs objects: 148.37M objects, 162 TiB usage: 276 TiB used, 572 TiB / 849 TiB avail pgs: 2786556/1119434807 objects misplaced (0.249%) 5559 active+clean 1 active+remapped+backfilling 1 active+clean+scrubbing+deep io: client: 212 MiB/s rd, 27 MiB/s wr, 4.04k op/s rd, 2.01k op/s wr recovery: 4.0 MiB/s, 4 objects/s progress: Global Recovery Event (2d) [===========================.] (remaining: 39m) So my question is, how can I adjust to make the osd data more evenly distributed? Thanks! Best Regards, Liang Zheng

1 1

Generated signurl is accessible from restricted IPs in bucket policy
by Aggelos Toumasis 24 Mar '23

24 Mar '23

Hi there, We noticed after creating a signurl that the bucket resources were accessible from IPs that were originally restricted from accessing them (using a bucket policy). Using the s3cmd utility we confirmed that the Policy is correctly applied and you can access it only for the allowed IPs. Is this an expected behavior or do we miss something? Thanks!

3 2

With Ceph Quincy, the "ceph" package does not include ceph-volume anymore
by Geert Kloosterman 24 Mar '23

24 Mar '23

Hi all, Until Ceph Pacific, installing just the "ceph" package was enough to get everything needed to deploy Ceph. However, with Quincy, ceph-volume was split off into its own package, and it is not automatically installed anymore. Here we can see it is not listed as a dependency: $ rpm -q --requires ceph binutils ceph-mds = 2:17.2.5-0.el8 ceph-mgr = 2:17.2.5-0.el8 ceph-mon = 2:17.2.5-0.el8 ceph-osd = 2:17.2.5-0.el8 rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(PayloadIsXz) <= 5.2-1 systemd Should I file a bug for this? Best regards, Geert Kloosterman

4 4

RGW can't create bucket
by Kamil Madac 24 Mar '23

24 Mar '23

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source -- Kamil Madac <https://kmadac.github.io/>

1 0

Unexpected ceph pool creation error with Ceph Quincy
by Geert Kloosterman 23 Mar '23

23 Mar '23

Hi all, I'm trying out Ceph Quincy (17.2.5) for the first time and I'm running into unexpected behavior of "ceph osd pool create". When not passing any pg_num and pgp_num values, I get the following error with Quincy: [root@gjk-ceph ~]# ceph osd pool create asdf Error ERANGE: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 I checked with Ceph Pacific (16.2.11) and there the extra arguments are not needed. I expected it would use osd_pool_default_pg_num and osd_pool_default_pgp_num as defined in my ceph.conf: [root@gjk-ceph ~]# ceph-conf -D | grep 'osd_pool_default_pg' osd_pool_default_pg_autoscale_mode = on osd_pool_default_pg_num = 8 osd_pool_default_pgp_num = 8 At least, this is what appears to be used with Pacific. Is this an intended change of behavior? I could not find anything related in the release notes. Best regards, Geert Kloosterman

2 5

Ceph performance problems
by Dominik Baack 23 Mar '23

23 Mar '23

Hi, we are currently testing out ways to increase Ceph performance because what we experience so far is very close to unusable. For the test cluster we utilizing 4 nodes with the following hardware data: Dual 200GBe Mellanox Ethernet 2x EPYC Rome 7302 16x 32GB 3200MHz ECC 9x 15.36TB Micron 9300 Pro For production this will be extended to all 8 nodes if it shows promising results. - Ceph was installed with cephadm. - MDS and ODS are located on the same nodes. - Mostly using stock config - Network performance tested with iperf3 seems fine, 26Gbits/s with -P4 on single port (details below). Close to 200Gbits with 10 parallel instances and servers. When testing a mounted CephFS on the working nodes in various configurations I only got <50MB/s for fuse mount and <270MB/s for kernel mounts. (dd command and output attached below) In addition ceph dashboard and our graphana monitoring reports packet loss on all relevant interfaces during load. Which does not occur during the normal iperf load tests or rsync/scp file transfer. Rados Bench shows performance around 2000MB/s which is not max performance of the SSDs but fine for us (details below). Why is the filesystem so slow compared to the individual components? Cheers Dominik Test details: ------------------------------------------------------------------------------------------------------ Some tests done on working nodes: Ceph mounted with ceph-fuse root@ml2ran10:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4,3 GB, 4,0 GiB) copied, 88,2933 s, 48,6 MB/s Ceph mounted with kernel driver: root@ml2ran06:/mnt/cephfs/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.0989 s, 267 MB/s Storage Node With fuse root@ml2rsn05:/mnt/ml2r_storage/backup# dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 53.9977 s, 79.5 MB/s Kernel mount: dd if=/dev/zero of=testfile bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.6726 s, 243 MB/s _______________________________________________________ Iperf3 iperf3 --zerocopy -n 10240M -P4 -c ml2ran08s0 -p 4701 -i 15 -b 200000000000 Connecting to host ml2ran08s0, port 4701 [ 5] local 129.217.31.180 port 43958 connected to 129.217.31.218 port 4701 [ 7] local 129.217.31.180 port 43960 connected to 129.217.31.218 port 4701 [ 9] local 129.217.31.180 port 43962 connected to 129.217.31.218 port 4701 [ 11] local 129.217.31.180 port 43964 connected to 129.217.31.218 port 4701 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 632 KBytes [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 522 KBytes [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 612 KBytes [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 430 KBytes [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 5] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 7] 0.00-3.21 sec 2.50 GBytes 6.70 Gbits/sec 0 sender [ 7] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [ 9] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 9] 0.00-3.21 sec 2.49 GBytes 6.67 Gbits/sec receiver [ 11] 0.00-3.21 sec 2.50 GBytes 6.69 Gbits/sec 0 sender [ 11] 0.00-3.21 sec 2.50 GBytes 6.67 Gbits/sec receiver [SUM] 0.00-3.21 sec 10.0 GBytes 26.8 Gbits/sec 0 sender [SUM] 0.00-3.21 sec 9.98 GBytes 26.7 Gbits/sec receiver _________________________________________________________________ Rados Bench on storage node: # rados bench -p testbench 10 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects Object prefix: benchmark_data_ml2rsn05_2829244 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 747 731 2923.84 2924 0.0178757 0.0216262 2 16 1506 1490 2979.71 3036 0.0308664 0.0213685 3 16 2267 2251 3000.99 3044 0.0259053 0.0212556 4 16 3058 3042 3041.62 3164 0.0227621 0.0209792 5 16 3850 3834 3066.8 3168 0.0130519 0.0208148 6 16 4625 4609 3072.26 3100 0.151371 0.0207904 7 16 5381 5365 3065.28 3024 0.0300368 0.0208345 8 16 6172 6156 3077.57 3164 0.0197728 0.0207714 9 16 6971 6955 3090.67 3196 0.0142751 0.0206786 10 14 7772 7758 3102.76 3212 0.0181034 0.020605 Total time run: 10.0179 Total writes made: 7772 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 3103.23 Stddev Bandwidth: 93.3676 Max bandwidth (MB/sec): 3212 Min bandwidth (MB/sec): 2924 Average IOPS: 775 Stddev IOPS: 23.3419 Max IOPS: 803 Min IOPS: 731 Average Latency(s): 0.020598 Stddev Latency(s): 0.00731743 Max latency(s): 0.151371 Min latency(s): 0.00966991 # rados bench -p testbench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 15 657 642 2567.32 2568 0.011104 0.022631 2 15 1244 1229 2456.9 2348 0.0115248 0.019485 3 16 1499 1483 1976.64 1016 0.00722983 0.0177887 4 15 1922 1907 1906.16 1696 0.0142242 0.0327382 5 15 2593 2578 2061.6 2684 0.011758 0.0301774 6 16 3142 3126 2083.23 2192 0.00915926 0.027478 7 16 3276 3260 1862.23 536 0.00824714 0.0267449 8 16 3606 3590 1794.43 1320 0.0118938 0.0350541 9 16 4293 4277 1900.32 2748 0.0301886 0.0330604 10 14 5003 4989 1995.04 2848 0.0389717 0.0314977 Total time run: 10.0227 Total reads made: 5003 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1996.67 Average IOPS: 499 Stddev IOPS: 202.3 Max IOPS: 712 Min IOPS: 134 Average Latency(s): 0.0314843 Max latency(s): 3.04463 Min latency(s): 0.00551523 # rados bench -p testbench 10 rand hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 15 15 0 0 0 - 0 1 15 680 665 2657.61 2660 0.00919807 0.0224833 2 15 1273 1258 2514.26 2372 0.00839656 0.0247125 3 16 1863 1847 2461.4 2356 0.00994467 0.0236565 4 16 2064 2048 2047.14 804 0.00809139 0.0223506 5 16 2064 2048 1637.79 0 - 0.0223506 6 16 2477 2461 1640.12 826 0.0286315 0.0383254 7 16 3102 3086 1762.89 2500 0.0267464 0.0349189 8 16 3513 3497 1748 1644 0.00890952 0.032269 9 16 3617 3601 1600 416 0.00626917 0.0316019 10 15 4014 3999 1599.18 1592 0.0461076 0.0393606 Total time run: 10.0481 Total reads made: 4014 Read size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1597.91 Average IOPS: 399 Stddev IOPS: 239.089 Max IOPS: 665 Min IOPS: 0 Average Latency(s): 0.0394035 Max latency(s): 3.00962 Min latency(s): 0.00449537

3 2

MDS host in OSD blacklist
by Frank Schilder 23 Mar '23

23 Mar '23

Hi all, we have an octopus v15.2.17 cluster and observe that one of our MDS hosts showed up in the OSD blacklist: # ceph osd blacklist ls 192.168.32.87:6801/3841823949 2023-03-22T10:08:02.589698+0100 192.168.32.87:6800/3841823949 2023-03-22T10:08:02.589698+0100 I see an MDS restart that might be related; see log snippets below. There are no clients running on this host, only OSDs and one MDS. What could be the reason for the blacklist entries? Thanks! Log snippets: Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:54 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: *** Caught signal (Aborted) ** Mar 21 10:07:54 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:54 ceph-23 journal: 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:54 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:54 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:54 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:54 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:54 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:54 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:54 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:54 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:54 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:54 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:54 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:54 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:54 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:54 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:54 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:54 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:54 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:54 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:54 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:54 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:54 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:54 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:54 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -1> 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:55 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:55 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: 0> 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:55 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:55 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:55 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:55 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:55 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -9999> 2023-03-21T10:07:54.980+0100 7f99e63d5700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f99e63d5700 time 2023-03-21T10:07:54.967936+0100 Mar 21 10:07:55 ceph-23 journal: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.17/rpm/el8/BUILD/ceph-15.2.17/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state == LOCK_XLOCK || state == LOCK_XLOCKDONE) Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f99f4a25b92] Mar 21 10:07:55 ceph-23 journal: 2: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 3: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 4: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 5: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 6: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 7: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 8: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 9: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 10: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: -9998> 2023-03-21T10:07:54.982+0100 7f99e63d5700 -1 *** Caught signal (Aborted) ** Mar 21 10:07:55 ceph-23 journal: in thread 7f99e63d5700 thread_name:MR_Finisher Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) Mar 21 10:07:55 ceph-23 journal: 1: (()+0x12ce0) [0x7f99f3605ce0] Mar 21 10:07:55 ceph-23 journal: 2: (gsignal()+0x10f) [0x7f99f2062a9f] Mar 21 10:07:55 ceph-23 journal: 3: (abort()+0x127) [0x7f99f2035e05] Mar 21 10:07:55 ceph-23 journal: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f99f4a25be3] Mar 21 10:07:55 ceph-23 journal: 5: (()+0x27ddac) [0x7f99f4a25dac] Mar 21 10:07:55 ceph-23 journal: 6: (MDCache::truncate_inode(CInode*, LogSegment*)+0x32c) [0x561bd623962c] Mar 21 10:07:55 ceph-23 journal: 7: (C_MDS_inode_update_finish::finish(int)+0x133) [0x561bd6210a83] Mar 21 10:07:55 ceph-23 journal: 8: (MDSContext::complete(int)+0x56) [0x561bd6422656] Mar 21 10:07:55 ceph-23 journal: 9: (MDSIOContextBase::complete(int)+0x39c) [0x561bd6422b5c] Mar 21 10:07:55 ceph-23 journal: 10: (MDSLogContextBase::complete(int)+0x44) [0x561bd6422cb4] Mar 21 10:07:55 ceph-23 journal: 11: (Finisher::finisher_thread_entry()+0x1a5) [0x7f99f4ab6a95] Mar 21 10:07:55 ceph-23 journal: 12: (()+0x81ca) [0x7f99f35fb1ca] Mar 21 10:07:55 ceph-23 journal: 13: (clone()+0x43) [0x7f99f204ddd3] Mar 21 10:07:55 ceph-23 journal: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Mar 21 10:07:55 ceph-23 journal: Mar 21 10:07:55 ceph-23 journal: reraise_fatal: default handler for signal 6 didn't terminate the process? Mar 21 10:07:58 ceph-23 dockerd-current: time="2023-03-21T10:07:58.119559277+01:00" level=warning msg="040c1e98a0669204e0e98bdbcdde893f8acf63444f3827358e663a13a2869478 cleanup: failed to unmount secrets: invalid argument" Mar 21 10:07:58 ceph-23 kernel: overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior. Mar 21 10:07:58 ceph-23 kernel: overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior. Mar 21 10:07:58 ceph-23 journal: 118 get_config /opt/ceph-container/bin/config.static.sh Mar 21 10:07:58 ceph-23 journal: 5 start_mds /opt/ceph-container/bin/start_mds.sh Mar 21 10:07:58 ceph-23 journal: 120 main /opt/ceph-container/bin/entrypoint.sh Mar 21 10:07:58 ceph-23 journal: 2023-03-21 10:07:58 /opt/ceph-container/bin/entrypoint.sh: static: does not generate config Mar 21 10:07:58 ceph-23 journal: 58 start_mds /opt/ceph-container/bin/start_mds.sh Mar 21 10:07:58 ceph-23 journal: 120 main /opt/ceph-container/bin/entrypoint.sh Mar 21 10:07:58 ceph-23 journal: 2023-03-21 10:07:58 /opt/ceph-container/bin/entrypoint.sh: SUCCESS Mar 21 10:07:58 ceph-23 journal: starting mds.ceph-23 at ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3

s3 compatible interface
by Marc 22 Mar '23

22 Mar '23

Anyone know of a s3 compatible interface that I can just run, and reads/writes files from a local file system and not from object storage?

13 18

S3 notification for backup
by Olivier Audry 22 Mar '23

22 Mar '23

Hello I would like to know if using the bucket notification system to http for backuping S3 bucket is a good move or not. Is someone already did it ? I got around 100to of little documents to backup and archive for legal purpose and dr each day. Many thanks for your help. oau

1 0

Moving From BlueJeans to Jitsi for Ceph meetings
by Mike Perez 22 Mar '23

22 Mar '23

Hi everyone, We have been using BlueJeans to meet and record some of our meetings that later get posted to our YouTube channel. Unfortunately, we have to figure out a new meeting platform due to Red Hat discontinuing BlueJeans by the end of this month. Google Meets is an option, but some users in other countries have trouble using Google's services. For some meetings, we have tried out Jitsi, and it works well, meets our requirements, and is free. Does anyone else have suggestions for another free meeting platform that provides recording capabilities? -- Mike Perez

6 6

Cephalocon Amsterdam 2023 Photographer Volunteer Help Needed
by Mike Perez 22 Mar '23

22 Mar '23

Hi everyone, To help with costs for Cephalocon Amsterdam 2023, we wanted to see if anyone would like to volunteer to help with photography for the event. A group of people would be ideal so that we have good coverage in the expo hall and sessions. If you're interested, please reply to me directly for more information. -- Mike Perez Community Manager Ceph Foundation

3 2

Re: Very slow backfilling/remapping of EC pool PGs
by Gauvain Pocentek 21 Mar '23

21 Mar '23

(adding back the list) On Tue, Mar 21, 2023 at 11:25 AM Joachim Kraftmayer < joachim.kraftmayer(a)clyso.com> wrote: > i added the questions and answers below. > > ___________________________________ > Best Regards, > Joachim Kraftmayer > CEO | Clyso GmbH > > Clyso GmbH > p: +49 89 21 55 23 91 2 > a: Loristraße 8 | 80335 München | Germany > w: https://clyso.com | e: joachim.kraftmayer(a)clyso.com > > We are hiring: https://www.clyso.com/jobs/ > --- > CEO: Dipl. Inf. (FH) Joachim Kraftmayer > Unternehmenssitz: Utting am Ammersee > Handelsregister beim Amtsgericht: Augsburg > Handelsregister-Nummer: HRB 25866 > USt. ID-Nr.: DE275430677 > > Am 21.03.23 um 11:14 schrieb Gauvain Pocentek: > > Hi Joachim, > > > On Tue, Mar 21, 2023 at 10:13 AM Joachim Kraftmayer < > joachim.kraftmayer(a)clyso.com> wrote: > >> Which Ceph version are you running, is mclock active? >> >> > We're using Quincy (17.2.5), upgraded step by step from Luminous if I > remember correctly. > > did you recreate the osds? if yes, at which version? > I actually don't remember all the history, but I think we added the HDD nodes while running Pacific. > > mlock seems active, set to high_client_ops profile. HDD OSDs have very > different settings for max capacity iops: > > osd.137 basic osd_mclock_max_capacity_iops_hdd > 929.763899 > osd.161 basic osd_mclock_max_capacity_iops_hdd > 4754.250946 > osd.222 basic osd_mclock_max_capacity_iops_hdd > 540.016984 > osd.281 basic osd_mclock_max_capacity_iops_hdd > 1029.193945 > osd.282 basic osd_mclock_max_capacity_iops_hdd > 1061.762870 > osd.283 basic osd_mclock_max_capacity_iops_hdd > 462.984562 > > We haven't set those explicitly, could they be the reason of the slow > recovery? > > i recommend to disable mclock for now, and yes we have seen slow recovery > caused by mclock. > Stupid question: how do you do that? I've looked through the docs but could only find information about changing the settings. > > > Bonus question: does ceph set that itself? > > yes and if you have a setup with HDD + SSD (db & wal) the discovery works > not in the right way. > Good to know! Gauvain > > Thanks! > > Gauvain > > > > >> Joachim >> >> ___________________________________ >> Clyso GmbH - Ceph Foundation Member >> >> Am 21.03.23 um 06:53 schrieb Gauvain Pocentek: >> > Hello all, >> > >> > We have an EC (4+2) pool for RGW data, with HDDs + SSDs for WAL/DB. This >> > pool has 9 servers with each 12 disks of 16TBs. About 10 days ago we >> lost a >> > server and we've removed its OSDs from the cluster. Ceph has started to >> > remap and backfill as expected, but the process has been getting slower >> and >> > slower. Today the recovery rate is around 12 MiB/s and 10 objects/s. All >> > the remaining unclean PGs are backfilling: >> > >> > data: >> > volumes: 1/1 healthy >> > pools: 14 pools, 14497 pgs >> > objects: 192.38M objects, 380 TiB >> > usage: 764 TiB used, 1.3 PiB / 2.1 PiB avail >> > pgs: 771559/1065561630 objects degraded (0.072%) >> > 1215899/1065561630 objects misplaced (0.114%) >> > 14428 active+clean >> > 50 active+undersized+degraded+remapped+backfilling >> > 18 active+remapped+backfilling >> > 1 active+clean+scrubbing+deep >> > >> > We've checked the health of the remaining servers, and everything looks >> > like (CPU/RAM/network/disks). >> > >> > Any hints on what could be happening? >> > >> > Thank you, >> > Gauvain >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >

2 2

Re: Ceph Bluestore tweaks for Bcache
by Matthias Ferdinand 21 Mar '23

21 Mar '23

ceph version: 17.2.0 on Ubuntu 22.04 non-containerized ceph from Ubuntu repos cluster started on luminous I have been using bcache on filestore on rotating disks for many years without problems. Now converting OSDs to bluestore, there are some strange effects. If I create the bcache device, set its rotational flag to '1', then do ceph-volume lvm create ... --crush-device-class=hdd the OSD comes up with the right parameters and much improved latency compared to OSD directly on /dev/sdX. ceph osd metatdata ... shows "bluestore_bdev_type": "hdd", "rotational": "1" But after reboot, bcache rotational flag is set '0' again, and the OSD now comes up with "rotational": "0" Latency immediately starts to increase (and continually increases over the next days, possibly due to accumulating fragmention). These wrong settings stay in place even if I stop the OSD, set the bcache rotational flag to '1' again and restart the OSD. I have found no way to get back to the original settings other than destroying and recreating the OSD. I guess I am just not seeing something obvious, like from where these settings get pulled at OSD startup. I even created udev rules to set bcache rotational=1 at boot time, before any ceph daemon starts, but it did not help. Something running after these rules reset the bcache rotationl flags back to 0. Haven't found the culprit yet, but not sure if it even matters. Are these OSD settings (bluestore_bdev_type, rotational) persisted somewhere and can they be edited and pinned? Alternatively, can I manually set and persist the relevant bluestore tunables (per OSD / per device class) so as to make the bcache rotational flag irrelevant after the OSD is first created? Regards Matthias On Fri, Apr 08, 2022 at 03:05:38PM +0300, Igor Fedotov wrote: > Hi Frank, > > in fact this parameter impacts OSD behavior at both build-time and during > regular operationing. It simply substitutes hdd/ssd auto-detection with > manual specification. And hence relevant config parameters are applied. If > e.g. min_alloc_size is persistent after OSD creation - it wouldn't be > updated. But if specific setting allows at run-time - it would be altered. > > So the proper usage would definitely be manual ssd/hdd mode selection before > the first OSD creation and keeping it in that mode along the whole OSD > lifecycle. But technically one can change the mode at any arbitrary point in > time which would result in run-rime setting being out-of-sync with creation > ones. With some unclear side-effects.. > > Please also note that this setting was orignally intended mostly for > development/testing purposes not regular usage. Hence it's flexible but > rather unsafe if used improperly. > > > Thanks, > > Igor > > On 4/7/2022 2:40 PM, Frank Schilder wrote: > > Hi Richard and Igor, > > > > are these tweaks required at build-time (osd prepare) only or are they required for every restart? > > > > Is this setting "bluestore debug enforce settings=hdd" in the ceph config data base or set somewhere else? How does this work if deploying HDD- and SSD-OSDs at the same time? > > > > Ideally, all these tweaks should be applicable and settable at creation time only without affecting generic settings (that is, at the ceph-volume command line and not via config side effects). Otherwise it becomes really tedious to manage these. > > > > For example, would the following work-flow apply the correct settings *permanently* across restarts: > > > > 1) Prepare OSD on fresh HDD with ceph-volume lvm batch --prepare ... > > 2) Assign dm_cache to logical OSD volume created in step 1 > > 3) Start OSD, restart OSDs, boot server ... > > > > I would assume that the HDD settings are burned into the OSD in step 1 and will be used in all future (re-)starts without the need to do anything despite the device being detected as non-rotational after step 2. Is this assumption correct? > > > > Thanks and best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Richard Bade <hitrich(a)gmail.com> > > Sent: 06 April 2022 00:43:48 > > To: Igor Fedotov > > Cc: Ceph Users > > Subject: [Warning Possible spam] [ceph-users] Re: Ceph Bluestore tweaks for Bcache > > > > Just for completeness for anyone that is following this thread. Igor > > added that setting in Octopus, so unfortunately I am unable to use it > > as I am still on Nautilus. > > > > Thanks, > > Rich > > > > On Wed, 6 Apr 2022 at 10:01, Richard Bade <hitrich(a)gmail.com> wrote: > > > Thanks Igor for the tip. I'll see if I can use this to reduce the > > > number of tweaks I need. > > > > > > Rich > > > > > > On Tue, 5 Apr 2022 at 21:26, Igor Fedotov <igor.fedotov(a)croit.io> wrote: > > > > Hi Richard, > > > > > > > > just FYI: one can use "bluestore debug enforce settings=hdd" config > > > > parameter to manually enforce HDD-related settings for a BlueStore > > > > > > > > > > > > Thanks, > > > > > > > > Igor > > > > > > > > On 4/5/2022 1:07 AM, Richard Bade wrote: > > > > > Hi Everyone, > > > > > I just wanted to share a discovery I made about running bluestore on > > > > > top of Bcache in case anyone else is doing this or considering it. > > > > > We've run Bcache under Filestore for a long time with good results but > > > > > recently rebuilt all the osds on bluestore. This caused some > > > > > degradation in performance that I couldn't quite put my finger on. > > > > > Bluestore osds have some smarts where they detect the disk type. > > > > > Unfortunately in the case of Bcache it detects as SSD, when in fact > > > > > the HDD parameters are better suited. > > > > > I changed the following parameters to match the HDD default values and > > > > > immediately saw my average osd latency during normal workload drop > > > > > from 6ms to 2ms. Peak performance didn't change really, but a test > > > > > machine that I have running a constant iops workload was much more > > > > > stable as was the average latency. > > > > > Performance has returned to Filestore or better levels. > > > > > Here are the parameters. > > > > > > > > > > ; Make sure that we use values appropriate for HDD not SSD - Bcache > > > > > gets detected as SSD > > > > > bluestore_prefer_deferred_size = 32768 > > > > > bluestore_compression_max_blob_size = 524288 > > > > > bluestore_deferred_batch_ops = 64 > > > > > bluestore_max_blob_size = 524288 > > > > > bluestore_min_alloc_size = 65536 > > > > > bluestore_throttle_cost_per_io = 670000 > > > > > > > > > > ; Try to improve responsiveness when some disks are fully utilised > > > > > osd_op_queue = wpq > > > > > osd_op_queue_cut_off = high > > > > > > > > > > Hopefully someone else finds this useful. > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > > > Igor Fedotov > > > > Ceph Lead Developer > > > > > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > > > > > croit GmbH, Freseniusstr. 31h, 81247 Munich > > > > CEO: Martin Verges - VAT-ID: DE310638492 > > > > Com. register: Amtsgericht Munich HRB 231263 > > > > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Igor Fedotov > Ceph Lead Developer > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Ceph Days India 2023 - Call for proposals
by Gaurav Sitlani 21 Mar '23

21 Mar '23

Hello Cephers, We're happy to share that we're organizing Ceph Day India on 5th May 2023 this year. The event is now sold out! If you missed getting a ticket, consider submitting the CFP and we'll provide a ticket if accepted. https://ceph.io/en/community/events/2023/ceph-days-india/ Please reach out to us if you need any help regarding the submissions. Thanks and regards, Gaurav Sitlani Ceph Community Ambassador

1 0

Changing os to ubuntu from centos 8
by Szabo, Istvan (Agoda) 21 Mar '23

21 Mar '23

Hi, I'd like to change the os to ubuntu 20.04.5 from my bare metal deployed octopus 15.2.14 on centos 8. On the first run I would go with octopus 15.2.17 just to not make big changes in the cluster. I've found couple of threads on the mailing list but those were containerized (like: Re: Upgrade/migrate host operating system for ceph nodes (CentOS/Rocky) or Re: Migrating CEPH OS looking for suggestions). Wonder what is the proper steps for this kind of migration? Do we need to start with mgr or mon or rgw or osd? Is it possible to reuse the osd with ceph-volume scan on the reinstalled machine? I'd stay with baremetal deployment and even maybe with octopus but I'm curious your advice. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Very slow backfilling/remapping of EC pool PGs
by Gauvain Pocentek 21 Mar '23

21 Mar '23

Hello all, We have an EC (4+2) pool for RGW data, with HDDs + SSDs for WAL/DB. This pool has 9 servers with each 12 disks of 16TBs. About 10 days ago we lost a server and we've removed its OSDs from the cluster. Ceph has started to remap and backfill as expected, but the process has been getting slower and slower. Today the recovery rate is around 12 MiB/s and 10 objects/s. All the remaining unclean PGs are backfilling: data: volumes: 1/1 healthy pools: 14 pools, 14497 pgs objects: 192.38M objects, 380 TiB usage: 764 TiB used, 1.3 PiB / 2.1 PiB avail pgs: 771559/1065561630 objects degraded (0.072%) 1215899/1065561630 objects misplaced (0.114%) 14428 active+clean 50 active+undersized+degraded+remapped+backfilling 18 active+remapped+backfilling 1 active+clean+scrubbing+deep We've checked the health of the remaining servers, and everything looks like (CPU/RAM/network/disks). Any hints on what could be happening? Thank you, Gauvain

2 1

Upgrade 16.2.10 --> 16.2.11 OSD "UPGRADE_REDEPLOY_DAEMON" failed
by Marco Pizzolo 20 Mar '23

20 Mar '23

Hello Everyone, We made the mistake of trying to patch to 16.2.11 from 16.2.10 which has been stable as we felt that 16.2.11 had been out for a while already. As luck would have it, we are having failure after failure with OSDs not upgrading successfully, and have 355 more OSDs to go. I'm pretty sure we're not alone on this and am wondering what others have done to address. Appreciate all suggestions. Thanks, Marco

1 1

Multiple instance_id and services for rbd-mirror daemon
by Aielli, Elia 20 Mar '23

20 Mar '23

Hi, I'm facing a strange issue and google doesn't seem to help me. I've a couple of clusters with Octopus v15.2.17, recently upgraded from 15.2.13 I had a rbd mirror service correctly working between the two clusters, i then updated, and after some days where all was ok, I've come to the situation where I have a single DAEMON, but multiple services (on different version tho), each one with its own instance_id, giving you some insight: rbd mirror pool status mypool --verbose health: WARNING daemon health: OK image health: WARNING images: 37 total 2 starting_replay 35 replaying DAEMONS service 123673101: instance_id: 127483700 client_id: admin hostname: DR-Host1 version: *15.2.13* leader: false health: OK service 123674106: instance_id: 127208040 client_id: admin hostname: DR-Host1 version: *15.2.13* leader: false health: OK service 123675375: instance_id: 124630539 client_id: admin hostname: DR-Host1 version: *15.2.13* leader: true health: OK service 124670331: instance_id: 127208013 client_id: backup hostname: DR-Host1 version: *15.2.17* leader: false health: OK As you can see one has the "leader" section as true, and the other are false, I'd like to delete the first 2 false ones, then delete the true one with 15.2.13 and hope the last one becomes true. Do any of you faced any similar issue or can help me in killing the wrong services? Let me know, thanks in advance! Elia

1 0

The release time of v16.2.12 is?
by Louis Koo 20 Mar '23

20 Mar '23

https://github.com/ceph/ceph/pull/47702?notification_referrer_id=NT_kwDOANW… This issue had been backported to pacific, and the release time of v16.2.12 is?

1 0

RBD latency
by Murilo Morais 17 Mar '23

17 Mar '23

Good evening everyone! Guys, what to expect latency for RBD images in a cluster with only HDD (36 HDDs)? Sometimes I see that the write latency is around 2-5 ms in some images even with very low IOPS and bandwidth while the read latency is around 0.2-0.7 ms. For a cluster with only HDD is this latency expected? Is there any parameter I can study to improve? What do you recommend for tuning in the OS? The latency between machines is always around 0.1ms, all connected via fiber optics. Thanks in advance!

3 2

radosgw SSE-C is not working (InvalidRequest)
by Boris Behrens 17 Mar '23

17 Mar '23

Hi, I try to evaluate SSE-C (so customer provides keys) for our object storages. We do not provide a KMS server. I've added "Access-Control-Allow-Headers" to the haproxy frontend. rspadd Access-Control-Allow-Headers... x-amz-server-side-encryption-customer-algorithm,\ x-amz-server-side-encryption-customer-key,\ x-amz-server-side-encryption-customer-key-MD5 I've also enabled "rgw_trust_forwarded_https = true" in the client section in the ceph.conf and restarted the RGW daemons. I now try to get it working, but I am not sure if I am doing it correctly. $ encKey=$(openssl rand -base64 32) $ md5Key=$(echo $encKey | md5sum | awk '{print $1}' | base64) $ aws s3api --endpoint=https://radosgw put-object \ --body ~/Downloads/TESTFILE \ --bucket test-bb-encryption \ --key TESTFILE \ --sse-customer-algorithm AES256 \ --sse-customer-key $encKey \ --sse-customer-key-md5 $md5Key This is what the RGW log gives me: 2023-03-17T10:55:55.465+0000 7f42bbe5f700 1 ====== starting new request req=0x7f448c185700 ===== 2023-03-17T10:55:55.469+0000 7f434df83700 1 ====== req done req=0x7f448c185700 op status=-2021 http_status=400 latency=3999985ns ====== 2023-03-17T10:55:55.469+0000 7f434df83700 1 beast: 0x7f448c185700: IPV6 - - [2023-03-17T10:55:55.469539+0000] "PUT /test-bb-encryption/TESTFILE HTTP/1.1" 400 221 - "aws-cli/2.4.18 Python/3.9.10 Darwin/22.3.0 source/x86_64 prompt/off command/s3api.put-object" - Maybe someone got a wroking example and is willing to share it with me, or did also encounter this problem and knows what to do? It's and octopus cluster. Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 1

tracker.ceph.com is slow
by Satoru Takeuchi 17 Mar '23

17 Mar '23

Hi, tracker.ceph.com seems to be quite slow recently. Since my colleagues feel so as well, this problem wouldn't be specific to me. Could you tel me if there is a plan to fix this problem near future? Thanks, Satoru

1 0

How to submit a bug report ?
by Patrick Vranckx 16 Mar '23

16 Mar '23

Hi, I suspect a bug in cephadm to configure ingress service for rgw. Our production server was upgraded from continuously from Luminous to Pacific. When configuring ingress service for rgw, the haproxy.cfg is incomplete. The same yaml file applied on our test cluster does the job. Regards, Patrick

2 1

Concerns about swap in ceph nodes
by sbryan Song 16 Mar '23

16 Mar '23

Hello, We have a 6-node ceph cluster, all of them have osd running and 3 of them (ceph-1 to ceph-3 )also has the ceph-mgr and ceph-mon. Here is the detailed configuration of each node (swap on ceph-1 to ceph-3 has been disabled after the alarm): # ceph-1 free -h total used free shared buff/cache available Mem: 187Gi 38Gi 5.4Gi 4.1Gi 143Gi 142Gi Swap: 0B 0B 0B # ceph-2 free -h total used free shared buff/cache available Mem: 187Gi 49Gi 2.6Gi 4.0Gi 135Gi 132Gi Swap: 0B 0B 0B # ceph-3 free -h total used free shared buff/cache available Mem: 187Gi 37Gi 4.6Gi 4.0Gi 145Gi 144Gi Swap: 0B 0B 0B # ceph-4 free -h total used free shared buff/cache available Mem: 251Gi 31Gi 8.3Gi 231Mi 211Gi 217Gi Swap: 124Gi 3.8Gi 121Gi # ceph-5 free -h total used free shared buff/cache available Mem: 251Gi 32Gi 14Gi 135Mi 204Gi 216Gi Swap: 124Gi 4.0Gi 121Gi # ceph-6 free -h total used free shared buff/cache available Mem: 251Gi 30Gi 16Gi 145Mi 204Gi 218Gi Swap: 124Gi 4.0Gi 121Gi We have configured swap space on all of them, for ceph-mgr nodes, we have 8G swap space and 128G swap configured for osd nodes, and our zabbix has monitored a swap over 50% usage for ceph-1 to ceph-3, but our available space are still around 140G against the total 187G. Just wondering whether the swap space is necessary when we have lots of memory available? Thanks very much for your answering.

3 3

Ceph NFS data - cannot read files, getattr returns NFS4ERR_PERM
by Wyll Ingersoll 16 Mar '23

16 Mar '23

ceph pacific 16.2.11 (cephadm managed) I have configured some NFS mounts from the ceph GUI from cephfs. We can mount the filesystems and view file/directory listings, but cannot read any file data. The permissions on the shares are RW. We mount from the client using "vers=4.1". Looking at debug logs from the container running nfs-ganesha, I see the following errors when trying to read a file's content: 15/03/2023 15:27:13 : epoch 6411e209 : gw01 : ganesha.nfsd-7[svc_8] complete_op :NFS4 :DEBUG :Status of OP_READ in position 2 = NFS4ERR_PERM, op response size is 7480 total response size is 7568 15/03/2023 15:27:13 : epoch 6411e209 : gw01 : ganesha.nfsd-7[svc_8] complete_nfs4_compound :NFS4 :DEBUG :End status = NFS4ERR_PERM lastindex = 3 Also, watching the TCP traffic, I see errors in the NFS protocol corresponding to these messages: 11:44:43.745570 IP xxx.747 > gw01.nfs: Flags [P.], seq 24184536:24184748, ack 11409577, win 602, options [nop,nop,TS val 342245425 ecr 2683489461], length 212: NFS request xid 156024373 208 getattr fh 0,1/53 11:44:43.745683 IP gw01.nfs > xxx.747: Flags [P.], seq 11409577:11409677, ack 24184748, win 3081, options [nop,nop,TS val 2683489461 ecr 342245425], length 100: NFS reply xid 156024373 reply ok 96 getattr ERROR: Operation not permitted So there appears to be a permissions problem where nfs-ganesha is not able to "getattr" on cephfs data. The export looks like this (read from rados): EXPORT { FSAL { name = "CEPH"; user_id = "nfs.cephfs.7"; filesystem = "cephfs"; secret_access_key = "xxx"; } export_id = 7; path = "/exports/nfs/foobar"; pseudo = "/foobar"; access_type = "RW"; squash = "no_root_squash"; attr_expiration_time = 0; security_label = false; protocols = 4; transports = "TCP"; } ceph auth permissions for the nfs.cephfs.7 client: [client.nfs.cephfs.7] key = xxx caps mds = "allow rw path=/exports/nfs/foobar" caps mon = "allow r" caps osd = "allow rw pool=.nfs namespace=cephfs, allow rw tag cephfs data=cephfs" Any suggestions?

2 8

Re: Stuck OSD service specification - can't remove
by Eugen Block 16 Mar '23

16 Mar '23

Hi, I tried to respond directly in the web ui of the mailing list but my message is queued for moderation. I just wanted to update a solution that worked for me when a service spec is stuck in a pending state, maybe this will help others in the same situation. While playing around with a test cluster I ended up with a "deleting" osd service spec. The SUSE team has an article [1] for this case, the following helped me resolve this issue. I had three different osd specs in place for the same three nodes: ---snip--- osd 3 <deleting> 3w nautilus2;nautilus3 osd.osd-hdd-ssd 3 2m ago 2w nautilus;nautilus2;nautilus3 osd.osd-hdd-ssd-mix 3 2m ago - <unmanaged> ---snip--- I replaced the "service_name" with the more suiting value ("osd.osd-hdd-ssd") in the unit.meta file of each OSD containing the invalid spec, then restarted each affected OSD. It probably wouldn't have been necessary but I wanted to see the effect immediately, so I failed over the mgr (ceph mgr fail), now I only have one valid osd spec. ---snip--- # before nautilus3:~ # grep service_name /var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/osd.3/unit.meta "service_name": "osd", # after nautilus3:~ # grep service_name /var/lib/ceph/201a2fbc-ce7b-44a3-9ed7-39427972083b/osd.3/unit.meta "service_name": "osd.osd-hdd-ssd", nautilus3:~ # ceph orch ls osd NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd-hdd-ssd 9 10m ago 2w nautilus;nautilus2;nautilus3 ---snip--- Regards, Eugen [1] https://www.suse.com/support/kb/doc/?id=000020667

1 0

External Auth (AssumeRoleWithWebIdentity) , STS by default, generic policies and isolation by ownership
by Christian Rohmann 16 Mar '23

16 Mar '23

Hello ceph-users, unhappy with the capabilities in regards to bucket access policies when using the Keystone authentication module I posted to this ML a while back - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/S2TV7GVFJT… In general I'd still like to hear how others are making use of external authentication and STS and what your experiences are in replacing e.g. Keystone authentication In the meantime we looked into OIDC authentication (via Keycloak) and the potentials there. While this works in general, AssumeRoleWithWebIdentity comes back with an STS token and that can be used to access S3 buckets, I am wondering about a few things: 1) How to enable STS for everyone (without user-individual policy to AssumeRole) In the documentation on STS (https://docs.ceph.com/en/quincy/radosgw/STS/#sts-in-ceph) and also STS-Lite (https://docs.ceph.com/en/quincy/radosgw/STSLite/#sts-lite) it's implied at one has to attach an dedicated policy to allow for STS to each user individually. This does not scale well with thousands of users. Also when using a federated / external authentication, there is no explicit user creation "A shadow user is created corresponding to every federated user. The user id is derived from the ‘sub’ field of the incoming web token." Is there a way to automatically have a role corresponding to each user that can be assumed via a OIDC token? So an implicit role that would allow for an externally authenticated user to have full access to S3 and all buckets owned? Looking at STS Lite documentation, it seems all the more natural to be able to allow keystone users to make use of STS. Is there any way to apply such an AssumeRole policy "globally" or for a whole set of users at the same time? I just found PR https://github.com/ceph/ceph/pull/44434 aiming to add policy variables such as ${aws:username} to allow for generic policies. But this is more about restricting bucket names or granting access to certain pattern of names. 2) Isolation in S3 Multi-Tenancy with external IdP (AssumeRoleWithWebIdentity), how does bucket ownership come into play? Following the question about generic policies for STS I am wondering about the role (no pun intended) that the bucket ownership or tenant play here? If one creates a role policy of e.g. {"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}} Would this allow someone assuming this role access to all, "*", buckets, or just those owned by the user that created this role policy? In case of Keystone auth the owner of a bucket is the project, not the individual (human) user. So this creates somewhat of a tenant which I'd want to isolate. 3) Allowing users to create their own roles and policies by default Is there a way to allow users to create their own roles and policies to use them by default? All the examples talk about the requirement for admin caps and individual setting of '--caps="user-policy=*'. If there was a default role + policy (question #1) that could be applied to externally authenticated users, I'd like for them to be able to create new roles and policies to grant access to their buckets to other users. Regards Christian

2 1

How to repair the OSDs while WAL/DB device breaks down
by Norman 16 Mar '23

16 Mar '23

hi, everyone, I have a question about repairing the broken WAL/DB device. I have a cluster with 8 OSDs, and 4 WAL/DB devices(1 OSD per WAL/DB device), and hwo can I repair the OSDs quickly if one WAL/DB device breaks down without rebuilding the them? Thanks.

2 4

Expression of Interest in Participating in GSoC 2023 with Your Team
by Arush Sharma 16 Mar '23

16 Mar '23

Dear Ceph Team, I hope this email finds you well. I am writing to express my keen interest in participating in the Google Summer of Code (GSoC) program 2023 with your team. I am a 3rd year B.tech student in Computer Science Engineering, with a strong passion for [specific area of interest related to the team's project(s)]. I have experience working in C++, and I believe that I can contribute significantly to your project by bringing my expertise, enthusiasm, and commitment. I have been following the GSoC program, and I understand the dedication and hard work required to complete a project successfully. Therefore, I am willing to commit my time and effort to meet the expectations and requirements of the program. I am open to learning new technologies and programming languages, and I believe that this opportunity will help me grow both personally and professionally. I have reviewed the list of your team's project ideas, and I am particularly interested in Disk Fragmentation Simulator. I would appreciate it if you could provide me with any additional information or resources that may be helpful to better understand the project requirements and goals. Thank you for taking the time to read my email, and I look forward to hearing back from you soon. Best regards, Arush Sharma

2 1

Upgrade 16.2.11 -> 17.2.0 failed
by bbk 16 Mar '23

16 Mar '23

Dear List, Today i was sucessfully upgrading with cephadm from 16.2.8 -> 16.2.9 -> 16.2.10 -> 16.2.11 Now i wanted to upgrade to 17.2.0 but after starting the upgrade with ``` # ceph orch upgrade start --ceph-version 17.2.0 ``` The orch manager module seems to be gone now and the upgrade don't seem to run. ``` # ceph orch upgrade status Error ENOENT: No orchestrator configured (try `ceph orch set backend`) # ceph orch set backend cephadm Error ENOENT: Module not found ``` During the failed upgrade all nodes had the 16.2.11 cephadm installed. Fortunately the cluster is still running... somehow. I installed the latest 17.2.X cephadm on all nodes and rebooted them nodes, but this didn't help. Does someone have a hint? Yours, bbk

4 7

Bluestore RocksDB Compression how to set
by Feng, Hualong 16 Mar '23

16 Mar '23

Hi all I want to study the effect of bluestore rocksdb compression on ceph and whether it is necessary to optimize it. But currently, bluestore rocksdb compression is disabled by default in ceph. I simply replaced the rocksdb compression algorithm, and then performed a 4KB rand read fio test, but the test results are not different. So I would like to ask you, is there any scenario where rocksdb needs to be compressed in ceph, and if so, what type of test is used to better demonstrate the impact of rocksdb compression on the upper layer of ceph. // create 10GB image ceph osd pool create rbd 128 rbd pool init rbd rbd create --size 10240 --image rbd/image --image-format 2 --thick-provision //fio [global] ioengine=rbd iodepth=128 rw=randwrite bs=4KB time_based=1 ramp_time=60s runtime=300s clientname=admin pool=rbd group_reporting buffer_compress_percentage=80 refill_buffers buffer_pattern=0xdeadbeef [volumes] rbdname=image numjobs=1 Thanks -Hualong

1 0

Ganesha NFS: Files disappearing
by Patrick Schlangen 15 Mar '23

15 Mar '23

Hi, today I saw a strange situation where files which were copied to a cephfs via Ganesha NFS (deployed via cephadm) disappeared from the NFS directory and then did not show up anymore until I restarted the ganesha instance. This could be observed on different NFS client hosts. While the files were not showing, they could still be seen via cephfs-shell, so I assume the issue is somewhere on the NFS side. Interestingly, the folder modification time (shown in ls) matched the time the file was copied to that folder. Any idea what could be the issue here? Unfortunately there were no interesting logs and I couldn't find a way yet to reproduce this. The ceph version in use is 17.2.5. Thanks, Patrick

2 2

10x more used space than expected
by Gaël THEROND 15 Mar '23

15 Mar '23

Hi everyone, I’ve got a quick question regarding one of our RadosGW bucket. This bucket is used to store docker registries, and the total amount of data we use is supposed to be 4.5Tb BUT it looks like ceph told us we rather use ~53Tb of data. One interesting thing is, this bucket seems to shard for unknown reason as it is supposed to be disabled by default, but even taking that into account we’re not supposed to see such a massive amount of additional data isn’t it? Here is the bucket stats of it: https://paste.opendev.org/show/bdWFRvNFtxyHnbPfXWu9/

4 8

Stuck OSD service specification - can't remove
by David Orman 15 Mar '23

15 Mar '23

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

3 4

Last day to sponsor Cephalocon Amsterdam 2023
by Mike Perez 14 Mar '23

14 Mar '23

Hi everyone, Today is the last day to sponsor Cephalocon Amsterdam 2023! I want to thank our current sponsors: Platinum: IBM Silver: 42on, Canonical Ubuntu, Clyso Startup: Koor Also, thank you to Clyso for their lanyard add-on and 42on's offsite attendee party. We are still short in covering the costs for the event, so I'm asking for contributors and members of the Ceph Foundation to consider applying today. https://events.linuxfoundation.org/cephalocon/sponsor/ Sponsor Prospectus: https://events.linuxfoundation.org/wp-content/uploads/2023/03/sponsor-ceph-… Please get in touch with us at sponsorships(a)ceph.foundation to get started. Thank you! -- Mike Perez

1 0

pg wait too long when osd restart
by yite gu 14 Mar '23

14 Mar '23

Hi all, osd_heartbeat_grace = 20 and osd_pool_default_read_lease_ratio = 0.8 by default, so, pg will wait 16s when osd restart in the worst case. This wait time is too long, client i/o can not be unacceptable. I think adjusting the osd_pool_default_read_lease_ratio to lower is a good way. Have any good suggestions about reduce pg wait time？ Best Regard Yite Gu

2 4

rbd on EC pool with fast and extremely slow writes/reads
by Andrej Filipcic 14 Mar '23

14 Mar '23

Hi, I have a problem on one of ceph clusters I do not understand. ceph 17.2.5 on 17 servers, 400 HDD OSDs, 10 and 25Gb/s NICs 3TB rbd image is on erasure coded 8+3 pool with 128pgs , xfs filesystem, 4MB objects in rbd image, mostly empy. I have created a bunch of 10G files, most of them were written with 1.5GB/s, few of them were really slow, ~10MB/s, a factor of 100. When reading these files back, the fast-written ones are read fast, ~2-2.5GB/s, the slowly-written are also extremely slow in reading, iotop shows between 1 and 30 MB/s reading speed. This does not happen at all on replicated images. There are some OSDs with higher apply/commit latency, eg 200ms, but there are no slow ops. The tests were done actually on proxmox vm with librbd, but the same happens with krbd, and on bare metal with mounted krbd as well. I have tried to check all OSDs for laggy drives, but they all look about the same. I have also copied entire image with "rados get...", object by object, the strange thing here is that most of objects were copied within 0.1-0.2s, but quite some took more than 1s. The cluster is quite busy with base traffic of ~1-2GB/s, so the speeds can vary due to that. But I would not expect a factor of 100 slowdown for some writes/reads with rbds. Any clues on what might be wrong or what else to check? I have another similar ceph cluster where everything looks fine. Best, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

3 3