February 2023 - ceph-users

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

Frequent calling monitor election
by Frank Schilder 04 May '23

04 May '23

Hi all, our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting: 2/9/23 4:42:30 PM[INF]overall HEALTH_OK 2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:40:00 PM[INF]overall HEALTH_OK 2/9/23 4:30:00 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:04 PM[INF]overall HEALTH_OK 2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:20:00 PM[INF]overall HEALTH_OK 2/9/23 4:10:00 PM[INF]overall HEALTH_OK 2/9/23 4:00:00 PM[INF]overall HEALTH_OK 2/9/23 3:50:00 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again? Thanks for any help! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 9

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

deploying Ceph using FQDN for MON / MDS Services
by Lokendra Rathour 21 Apr '23

21 Apr '23

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

7 20

ceph/daemon stable tag
by Jonas Nemeikšis 15 Apr '23

15 Apr '23

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

2 6

avg apply latency went up after update from octopus to pacific
by Boris Behrens 02 Apr '23

02 Apr '23

Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms. All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value? Cheers Boris

10 27

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

Generated signurl is accessible from restricted IPs in bucket policy
by Aggelos Toumasis 24 Mar '23

24 Mar '23

Hi there, We noticed after creating a signurl that the bucket resources were accessible from IPs that were originally restricted from accessing them (using a bucket policy). Using the s3cmd utility we confirmed that the Policy is correctly applied and you can access it only for the allowed IPs. Is this an expected behavior or do we miss something? Thanks!

3 2

s3 compatible interface
by Marc 22 Mar '23

22 Mar '23

Anyone know of a s3 compatible interface that I can just run, and reads/writes files from a local file system and not from object storage?

13 18

Re: Ceph Bluestore tweaks for Bcache
by Matthias Ferdinand 21 Mar '23

21 Mar '23

ceph version: 17.2.0 on Ubuntu 22.04 non-containerized ceph from Ubuntu repos cluster started on luminous I have been using bcache on filestore on rotating disks for many years without problems. Now converting OSDs to bluestore, there are some strange effects. If I create the bcache device, set its rotational flag to '1', then do ceph-volume lvm create ... --crush-device-class=hdd the OSD comes up with the right parameters and much improved latency compared to OSD directly on /dev/sdX. ceph osd metatdata ... shows "bluestore_bdev_type": "hdd", "rotational": "1" But after reboot, bcache rotational flag is set '0' again, and the OSD now comes up with "rotational": "0" Latency immediately starts to increase (and continually increases over the next days, possibly due to accumulating fragmention). These wrong settings stay in place even if I stop the OSD, set the bcache rotational flag to '1' again and restart the OSD. I have found no way to get back to the original settings other than destroying and recreating the OSD. I guess I am just not seeing something obvious, like from where these settings get pulled at OSD startup. I even created udev rules to set bcache rotational=1 at boot time, before any ceph daemon starts, but it did not help. Something running after these rules reset the bcache rotationl flags back to 0. Haven't found the culprit yet, but not sure if it even matters. Are these OSD settings (bluestore_bdev_type, rotational) persisted somewhere and can they be edited and pinned? Alternatively, can I manually set and persist the relevant bluestore tunables (per OSD / per device class) so as to make the bcache rotational flag irrelevant after the OSD is first created? Regards Matthias On Fri, Apr 08, 2022 at 03:05:38PM +0300, Igor Fedotov wrote: > Hi Frank, > > in fact this parameter impacts OSD behavior at both build-time and during > regular operationing. It simply substitutes hdd/ssd auto-detection with > manual specification. And hence relevant config parameters are applied. If > e.g. min_alloc_size is persistent after OSD creation - it wouldn't be > updated. But if specific setting allows at run-time - it would be altered. > > So the proper usage would definitely be manual ssd/hdd mode selection before > the first OSD creation and keeping it in that mode along the whole OSD > lifecycle. But technically one can change the mode at any arbitrary point in > time which would result in run-rime setting being out-of-sync with creation > ones. With some unclear side-effects.. > > Please also note that this setting was orignally intended mostly for > development/testing purposes not regular usage. Hence it's flexible but > rather unsafe if used improperly. > > > Thanks, > > Igor > > On 4/7/2022 2:40 PM, Frank Schilder wrote: > > Hi Richard and Igor, > > > > are these tweaks required at build-time (osd prepare) only or are they required for every restart? > > > > Is this setting "bluestore debug enforce settings=hdd" in the ceph config data base or set somewhere else? How does this work if deploying HDD- and SSD-OSDs at the same time? > > > > Ideally, all these tweaks should be applicable and settable at creation time only without affecting generic settings (that is, at the ceph-volume command line and not via config side effects). Otherwise it becomes really tedious to manage these. > > > > For example, would the following work-flow apply the correct settings *permanently* across restarts: > > > > 1) Prepare OSD on fresh HDD with ceph-volume lvm batch --prepare ... > > 2) Assign dm_cache to logical OSD volume created in step 1 > > 3) Start OSD, restart OSDs, boot server ... > > > > I would assume that the HDD settings are burned into the OSD in step 1 and will be used in all future (re-)starts without the need to do anything despite the device being detected as non-rotational after step 2. Is this assumption correct? > > > > Thanks and best regards, > > ================= > > Frank Schilder > > AIT Risø Campus > > Bygning 109, rum S14 > > > > ________________________________________ > > From: Richard Bade <hitrich(a)gmail.com> > > Sent: 06 April 2022 00:43:48 > > To: Igor Fedotov > > Cc: Ceph Users > > Subject: [Warning Possible spam] [ceph-users] Re: Ceph Bluestore tweaks for Bcache > > > > Just for completeness for anyone that is following this thread. Igor > > added that setting in Octopus, so unfortunately I am unable to use it > > as I am still on Nautilus. > > > > Thanks, > > Rich > > > > On Wed, 6 Apr 2022 at 10:01, Richard Bade <hitrich(a)gmail.com> wrote: > > > Thanks Igor for the tip. I'll see if I can use this to reduce the > > > number of tweaks I need. > > > > > > Rich > > > > > > On Tue, 5 Apr 2022 at 21:26, Igor Fedotov <igor.fedotov(a)croit.io> wrote: > > > > Hi Richard, > > > > > > > > just FYI: one can use "bluestore debug enforce settings=hdd" config > > > > parameter to manually enforce HDD-related settings for a BlueStore > > > > > > > > > > > > Thanks, > > > > > > > > Igor > > > > > > > > On 4/5/2022 1:07 AM, Richard Bade wrote: > > > > > Hi Everyone, > > > > > I just wanted to share a discovery I made about running bluestore on > > > > > top of Bcache in case anyone else is doing this or considering it. > > > > > We've run Bcache under Filestore for a long time with good results but > > > > > recently rebuilt all the osds on bluestore. This caused some > > > > > degradation in performance that I couldn't quite put my finger on. > > > > > Bluestore osds have some smarts where they detect the disk type. > > > > > Unfortunately in the case of Bcache it detects as SSD, when in fact > > > > > the HDD parameters are better suited. > > > > > I changed the following parameters to match the HDD default values and > > > > > immediately saw my average osd latency during normal workload drop > > > > > from 6ms to 2ms. Peak performance didn't change really, but a test > > > > > machine that I have running a constant iops workload was much more > > > > > stable as was the average latency. > > > > > Performance has returned to Filestore or better levels. > > > > > Here are the parameters. > > > > > > > > > > ; Make sure that we use values appropriate for HDD not SSD - Bcache > > > > > gets detected as SSD > > > > > bluestore_prefer_deferred_size = 32768 > > > > > bluestore_compression_max_blob_size = 524288 > > > > > bluestore_deferred_batch_ops = 64 > > > > > bluestore_max_blob_size = 524288 > > > > > bluestore_min_alloc_size = 65536 > > > > > bluestore_throttle_cost_per_io = 670000 > > > > > > > > > > ; Try to improve responsiveness when some disks are fully utilised > > > > > osd_op_queue = wpq > > > > > osd_op_queue_cut_off = high > > > > > > > > > > Hopefully someone else finds this useful. > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > > > Igor Fedotov > > > > Ceph Lead Developer > > > > > > > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > > > > > > > croit GmbH, Freseniusstr. 31h, 81247 Munich > > > > CEO: Martin Verges - VAT-ID: DE310638492 > > > > Com. register: Amtsgericht Munich HRB 231263 > > > > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Igor Fedotov > Ceph Lead Developer > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Stuck OSD service specification - can't remove
by David Orman 15 Mar '23

15 Mar '23

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

3 4

Undo "radosgw-admin bi purge"
by Robert Sander 13 Mar '23

13 Mar '23

Hi, There is an operation "radosgw-admin bi purge" that removes all bucket index objects for one bucket in the rados gateway. What is the undo operation for this? After this operation the bucket cannot be listed or removed any more. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

4 9

mds readonly, mds all down
by kreept.sama＠gmail.com 06 Mar '23

06 Mar '23

Hello. We trying to resolve some issue with ceph. Our openshift cluster is blocked and we tried do almost all. Actual state is: MDS_ALL_DOWN: 1 filesystem is offline MDS_DAMAGE: 1 mds daemon damaged FS_DEGRADED: 1 filesystem is degraded MON_DISK_LOW: mon be is low on available space RECENT_CRASH: 1 daemons have recently crashed We try to perform cephfs-journal-tool --rank=gml-okd-cephfs:all event recover_dentries summary cephfs-journal-tool --rank=gml-okd-cephfs:all journal reset cephfs-table-tool gml-okd-cephfs:all reset session ceph mds repaired 0 ceph config rm mds mds_verify_scatter ceph config rm mds mds_debug_scatterstat ceph tell gml-okd-cephfs scrub start / recursive repair force After these commands, mds rises but an error appears: MDS_READ_ONLY: 1 MDSs are read only We also tried to create new fs with new metadata pool, delete and recreate old fs with same name with old\new metadatapool. We got rid of the errors, but the Openshift cluster did not want to work with the old persistence volumes. The pods wrote an error that they could not find it, while it was present and moreover, this volume was associated with pvc. Now we have rolled back the cluster and are trying to remove the mds error. Any ideas what to try? Thanks

2 2

ceph quincy nvme drives displayed in device list sata ssd not displayed
by Chris Brown 03 Mar '23

03 Mar '23

Hello, Setting up first ceph cluster in lab. Rocky 8.6 Ceph quincy Using curl install method Following cephadm deployment steps Everything works as expected except ceph orch device ls --refresh Only displays nvme devices and not the sata ssds on the ceph host. Tried sgdisk --zap-all /dev/sda wipefs -a /dev/sda Adding sata osd manually I get: ceph orch daemon add osd ceph-a:data_devices=/dev/sda Created no osd(s) on host ceph-a; already created? nvme osd gets added without issue. I have looked in the volume log on the node and monitor log on the admin server and have not seen anything that seems like an obvious clue. I can see commands running successfully against /dev/sda in the logs. Ideas? Thanks, cb

1 1

ceph 16.2.10 - misplaced object after changing crush map only setting hdd class
by xadhoom76＠gmail.com 02 Mar '23

02 Mar '23

Hi to all and thanks for sharing your experience on ceph ! We have an easy setup with 9 osd all hdd and 3 nodes, 3 osd for each node. We started the cluster to test how it works with hdd with default and easy bootstrap . Then we decide to add ssd and create a pool to use only ssd. In order to have pools on hdd and pools on ssd only we edited the crushmap to add class hdd We do not enter anything about ssd till now, nor disk or rules only add the class map to the default rule. So i show you the rules before introducing class hdd # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure2_1 { id 2 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure-pool.meta { id 3 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } rule erasure-pool.data { id 4 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default step chooseleaf indep 0 type host step emit } And here is the after # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type host step emit } rule erasure-code { id 1 type erasure min_size 3 max_size 4 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure2_1 { id 2 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure-pool.meta { id 3 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } rule erasure-pool.data { id 4 type erasure min_size 3 max_size 3 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } Just doing this triggered the misplaced of all pgs bind to EC pool. Is that correct ? and why ? Best regards Alessandro Bolgia

2 2

How do I troubleshoot radosgw errors STS?
by mat＠hazmat.dev 02 Mar '23

02 Mar '23

I've setup RadosGW with STS ontop of my ceph cluster. It works great and fine but I'm also trying to setup authentication with an OpenIDConnect provider. I'm have a hard time troubleshooting issues because the radosgw log file doesn't have much information in it. For example when I try to use the `sts:AssumeRoleWithWebIdentity` API it fails with `{'Code': 'AccessDenied', ...}` and all I see is the beat log showing an HTTP 403. Is there a way to enable more verbose logging so I can see what is failing and why I'm getting certain errors with STS, S3, or IAM apis? My ceph.conf looks like this for each node (mildly redacted): ``` [client.radosgw.pve4] host = pve4 keyring = /etc/pve/priv/ceph.client.radosgw.keyring log file = /var/log/ceph/client.radosgw.$host.log rgw_dns_name = s3.lab rgw_frontends = beast endpoint=0.0.0.0:7480 ssl_endpoint=0.0.0.0:443 ssl_certificate=/etc/pve/priv/ceph/s3.lab.crt ssl_private_key=/etc/pve/priv/ceph/s3.lab.key rgw_sts_key = 1111111111111111 rgw_s3_auth_use_sts = true rgw_enable_apis = s3, s3website, admin, sts, iam ```

2 3

Next quincy release (17.2.6)
by Yuri Weinstein 01 Mar '23

01 Mar '23

Hello We are planning to start QE validation release next week. If you have PRs that are to be part of it, please let us know by adding "needs-qa" for 'quincy' milestone ASAP. Thx YuriW

3 3

Ceph OSD imbalance and performance
by Dave Ingram 28 Feb '23

28 Feb '23

Hello, Our ceph cluster performance has become horrifically slow over the past few months. Nobody here is terribly familiar with ceph and we're inheriting this cluster without much direction. Architecture: 40Gbps QDR IB fabric between all ceph nodes and our ovirt VM hosts. 11 OSD nodes with a total of 163 OSDs. 14 pools, 3616 PGs, 1.19PB total capacity. Ceph versions: { "mon": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 3 }, "mgr": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 3 }, "osd": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 118, "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 22, "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 19 }, "mds": {}, "overall": { "ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)": 124, "ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)": 22, "ceph version 12.2.13 (584a20eb0237c657dc0567da126be145106aa47e) luminous (stable)": 19 } } The majority of disks are spindles but there are also NVMe SSDs. There is a lot of variability in drive sizes - two different sets of admins added disks sized between 6TB and 16TB and I suspect this and imbalanced weighting is to blame. Performance on the ovirt VMs can dip as low as several *kilobytes* per-second (!) on reads and a few MB/sec on writes. There are also several scrub errors. In short, it's a complete wreck. STATUS: [root@ceph-admin davei]# ceph -s cluster: id: 1b8d958c-e50b-40ef-a681-16cfeb9390b8 health: HEALTH_ERR 3 scrub errors Possible data damage: 3 pgs inconsistent services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 mgr: ceph3(active), standbys: ceph2, ceph1 osd: 163 osds: 159 up, 158 in data: pools: 14 pools, 3616 pgs objects: 46.28M objects, 174TiB usage: 527TiB used, 694TiB / 1.19PiB avail pgs: 3609 active+clean 4 active+clean+scrubbing+deep 3 active+clean+inconsistent io: client: 74.3MiB/s rd, 96.0MiB/s wr, 3.85kop/s rd, 3.68kop/s wr --- HEALTH: [root@ceph-admin davei]# ceph health detail HEALTH_ERR 3 scrub errors; Possible data damage: 3 pgs inconsistent OSD_SCRUB_ERRORS 3 scrub errors PG_DAMAGED Possible data damage: 3 pgs inconsistent pg 2.8a is active+clean+inconsistent, acting [13,152,127] pg 2.ce is active+clean+inconsistent, acting [145,13,152] pg 2.e8 is active+clean+inconsistent, acting [150,162,42] --- CEPH OSD DF: (not going to paste that all in here): https://pastebin.com/CNW5RKWx What else am I missing in terms of what to share with you all? Any advice on how we should 'reweight' these to get the performance to improve? Thanks all, -Dave

4 6

Do not use SSDs with (small) SLC cache
by Michael Wodniok 28 Feb '23

28 Feb '23

Hi all, digging around debugging, why our (small: 10 Hosts/~60 OSDs) cluster is so slow even while recovering I found out one of our key issues are some SSDs with SLC cache (in our case Samsung SSD 870 EVO) - which we just recycled from other use cases in the hope to speed up our mainly hdd based cluster. We know it's a little bit random which objects get accelerated when not used as cache. However the opposite was the case. These type's ssds are only fast when operating in their SLC cache, which is only several Gigabytes in a multi-TB ssd [1]. When doing a big write or a backfill onto these SSDs we got really low IO-rates (around 10 MB/s even with 4M-objects). But it got even worse. Disclaimer: This is my view as a user, maybe a more technically involved person is able to correct me. Cause seems to be the mclock-scheduler which measures the iops an osd is able to do. As in the blog measured [2], this is usually a good thing as there is done some profiling and queing is done different. But in our case the osd_mclock_max_capacity_iops_ssd for most of the corresponding osds was very low. But not for everyone. I assume that it depends when mclock-scheduler measured the iops capacity. That led to a broken scheduling where backfills were at low speed and the ssd itself had nearly no disk usage because it was operating in it's cache again and could work faster. That issue could be solved by switching back to wpq scheduler for the affected SSDs. This scheduler seems to just queue up ios without throttling because of maximum iops reached. Now we see a still bad IO situation because of the slow SSDs but at least they are operating at their maximum (having typical settings like osd_recovery_max_active and osd_recovery_sleep* tuned). We are going to replace the SSDs to hopefully more consistent performing ones (even if their peak performance would be not as good). I hope this may help somebody in the future when being stuck in low performance recoverys. Refs: [1] https://www.tomshardware.com/reviews/samsung-870-evo-sata-ssd-review-the-be… [2] https://ceph.io/en/news/blog/2022/mclock-vs-wpq-testing-with-background-ops… Happy Storing! Michael Wodniok -- Michael Wodniok M.Sc. WorNet AG Bürgermeister-Graf-Ring 28 82538 Geretsried Simply42 und SecuMail sind Marken der WorNet AG. http://www.wor.net/ Handelsregister Amtsgericht München (HRB 129882) Vorstand: Christian Eich Aufsichtsratsvorsitzender: Dirk Steinkopf

7 8

CephFS Kernel Mount Options Without Mount Helper
by Shawn Weeks 28 Feb '23

28 Feb '23

I’m trying to find documentation for which mount options are supported directly by the kernel module. For example in the kernel module included in Rocky Linux 8 and 9 the secretfile option isn’t supported even though the documentation seems to imply it is. It seems like the documentation assumes you’ll always be using the mount.ceph helper and I’m trying to find out what options are supported if you don’t have mount.ceph helper. Thanks Shawn

1 1

[RGW] Rebuilding a non master zone
by Gilles Mocellin 28 Feb '23

28 Feb '23

Hi Cephers, I have large OMAP objects on one of my cluster (certainly due to a big bucket deletion, and things not completely purged). Since there is no tool to either reconstruct index from data or purge unused index, I thought I can use mutlisite replication. As I am in a multisite configuration, and the cluster is not in the master zone, will all the data be recovered from the master zone if I stop radosgw, delete RGW index and data pools, and restart radosgw ? Or will it definitively not be so simple ? The data pool reports 8TB used. Even if it works, it will take ages... If someone has another idea to remove those large OMAP objects... I've seen several times that question on the mailing list, but never saw a response that works or was adapted for my use case. The rgw-orphan-list script could be a solution, but too long to run on my cluster. And still, I have to know if I really can delete objects, and I have no clue...

1 0

OSD upgrade problem nautilus->octopus - snap_mapper upgrade stuck
by Jan Pekař - Imatic 28 Feb '23

28 Feb '23

Hi all, I have problem upgrading nautilus to octopus on my OSD. Upgrade mon and mgr was OK and first OSD stuck on 2023-01-12T09:25:54.122+0100 7f49ff3eae00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) and there were no activity after that for more than 48 hours. No disk activity. I restarted OSD many times and nothing changed. It is old, filestore OSD based on XFS filesystem. Is upgrade to snap mapper 2 reliable? What is OSD waiting for? Can I start OSD without upgrade and get cluster healthy with old snap structure? Or should I skip octopus upgrade and go to pacific directly (some bug backport is missing?). Thank you for help, I'm sending some logs below.. Log shows 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 ceph version 15.2.17 (694d03a6f6c6e9f814446223549caf9a9f60dba0) octopus (stable), process ceph-osd, pid 2566563 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 pidfile_write: ignore empty --pid-file 2023-01-09T19:12:49.499+0100 7f41f60f1e00 -1 missing 'type' file, inferring filestore from current/ dir 2023-01-09T19:12:49.531+0100 7f41f60f1e00 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2023-01-09T19:12:49.531+0100 7f41f60f1e00 -1 Falling back to public interface 2023-01-09T19:12:49.871+0100 7f41f60f1e00 0 load: jerasure load: lrc load: isa 2023-01-09T19:12:49.875+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice() is disabled via 'filestore splice' config option 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf 2023-01-09T19:12:50.015+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) start omap initiation 2023-01-09T19:12:50.079+0100 7f41f60f1e00 1 leveldb: Recovering log #165531 2023-01-09T19:12:50.083+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: started 2023-01-09T19:12:50.235+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: 1598 bytes OK 2023-01-09T19:12:50.583+0100 7f41f60f1e00 1 leveldb: Delete type=0 #165531 2023-01-09T19:12:50.615+0100 7f41f60f1e00 1 leveldb: Delete type=3 #165529 2023-01-09T19:12:51.339+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) mount(1861): enabling WRITEAHEAD journal mode: checkpoint is not enabled 2023-01-09T19:12:51.379+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.931+0100 7f41f60f1e00 -1 journal do_read_entry(243675136): bad header magic 2023-01-09T19:12:51.939+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.943+0100 7f41f60f1e00 1 filestore(/var/lib/ceph/osd/ceph-0) upgrade(1466) 2023-01-09T19:12:52.015+0100 7f41f60f1e00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) lsof shows COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ceph-osd 225860 ceph cwd DIR 9,127 4096 2 / ceph-osd 225860 ceph rtd DIR 9,127 4096 2 / ceph-osd 225860 ceph txt REG 9,127 31762544 5021 /usr/bin/ceph-osd ceph-osd 225860 ceph mem REG 8,70 2147237 68104224 /var/lib/ceph/osd/ceph-0/current/omap/165546.ldb ceph-osd 225860 ceph mem REG 8,70 2147792 68104190 /var/lib/ceph/osd/ceph-0/current/omap/165545.ldb ceph-osd 225860 ceph mem REG 8,70 2147689 68104240 /var/lib/ceph/osd/ceph-0/current/omap/165466.ldb ceph-osd 225860 ceph mem REG 8,70 2142721 68102679 /var/lib/ceph/osd/ceph-0/current/omap/165544.ldb ceph-osd 225860 ceph mem REG 8,70 2142677 68104239 /var/lib/ceph/osd/ceph-0/current/omap/165465.ldb ceph-osd 225860 ceph mem REG 8,70 2144979 68078254 /var/lib/ceph/osd/ceph-0/current/omap/165543.ldb ceph-osd 225860 ceph mem REG 8,70 2143705 68163491 /var/lib/ceph/osd/ceph-0/current/omap/165526.ldb ceph-osd 225860 ceph mem REG 8,70 2141468 68163492 /var/lib/ceph/osd/ceph-0/current/omap/165527.ldb ceph-osd 225860 ceph mem REG 8,70 145986 68018644 /var/lib/ceph/osd/ceph-0/current/omap/165541.ldb ceph-osd 225860 ceph mem REG 8,70 2143434 68163490 /var/lib/ceph/osd/ceph-0/current/omap/165525.ldb ceph-osd 225860 ceph mem REG 8,70 2136002 68122351 /var/lib/ceph/osd/ceph-0/current/omap/165472.ldb ceph-osd 225860 ceph mem REG 8,70 1965262 68119647 /var/lib/ceph/osd/ceph-0/current/omap/165467.ldb ceph-osd 225860 ceph mem REG 8,70 2145206 68104229 /var/lib/ceph/osd/ceph-0/current/omap/165464.ldb ceph-osd 225860 ceph mem REG 8,70 61600 68002130 /var/lib/ceph/osd/ceph-0/current/omap/165536.ldb ceph-osd 225860 ceph mem REG 8,70 352689 67945734 /var/lib/ceph/osd/ceph-0/current/omap/165530.ldb ceph-osd 225860 ceph mem REG 8,70 82500 67480519 /var/lib/ceph/osd/ceph-0/current/omap/165539.ldb ceph-osd 225860 ceph mem REG 9,127 189200 2878 /usr/lib/ceph/erasure-code/libec_isa.so ceph-osd 225860 ceph mem REG 9,127 2021136 3033 /usr/lib/ceph/erasure-code/libec_lrc.so ceph-osd 225860 ceph mem REG 9,127 360544 2942 /usr/lib/ceph/erasure-code/libec_jerasure.so ceph-osd 225860 ceph mem REG 9,127 55792 15227 /lib/x86_64-linux-gnu/libnss_files-2.28.so ceph-osd 225860 ceph mem REG 9,127 22368 133997 /usr/lib/x86_64-linux-gnu/liburcu-common.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 42976 133996 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 51128 149821 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0 ceph-osd 225860 ceph mem REG 8,70 33977 68116674 /var/lib/ceph/osd/ceph-0/current/omap/165547.ldb ceph-osd 225860 ceph DEL REG 0,19 64261192 /[aio] ceph-osd 225860 ceph DEL REG 0,19 64261191 /[aio] ceph-osd 225860 ceph mem REG 9,127 158400 2523 /lib/x86_64-linux-gnu/liblzma.so.5.2.4 ceph-osd 225860 ceph mem REG 9,127 135064 12635 /lib/x86_64-linux-gnu/libnl-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 488272 133248 /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 35808 15263 /lib/x86_64-linux-gnu/librt-2.28.so ceph-osd 225860 ceph mem REG 9,127 55512 133042 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1 ceph-osd 225860 ceph mem REG 9,127 30776 16463 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 ceph-osd 225860 ceph mem REG 9,127 1820400 15070 /lib/x86_64-linux-gnu/libc-2.28.so ceph-osd 225860 ceph mem REG 9,127 100712 7687 /lib/x86_64-linux-gnu/libgcc_s.so.1 ceph-osd 225860 ceph mem REG 9,127 1579448 15202 /lib/x86_64-linux-gnu/libm-2.28.so ceph-osd 225860 ceph mem REG 9,127 1570256 133576 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 ceph-osd 225860 ceph mem REG 9,127 112800 133289 /usr/lib/x86_64-linux-gnu/librdmacm.so.1.1.22.1 ceph-osd 225860 ceph mem REG 9,127 104496 133287 /usr/lib/x86_64-linux-gnu/libibverbs.so.1.5.22.1 ceph-osd 225860 ceph mem REG 9,127 149704 5527 /lib/x86_64-linux-gnu/libudev.so.1.6.13 ceph-osd 225860 ceph mem REG 9,127 146968 15251 /lib/x86_64-linux-gnu/libpthread-2.28.so ceph-osd 225860 ceph mem REG 9,127 3044224 135371 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 ceph-osd 225860 ceph mem REG 9,127 14592 15128 /lib/x86_64-linux-gnu/libdl-2.28.so ceph-osd 225860 ceph mem REG 9,127 93000 15255 /lib/x86_64-linux-gnu/libresolv-2.28.so ceph-osd 225860 ceph mem REG 9,127 301288 134118 /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3 ceph-osd 225860 ceph mem REG 9,127 14184 134682 /usr/lib/x86_64-linux-gnu/libaio.so.1.0.1 ceph-osd 225860 ceph mem REG 9,127 117184 2738 /lib/x86_64-linux-gnu/libz.so.1.2.11 ceph-osd 225860 ceph mem REG 9,127 121184 136027 /usr/lib/x86_64-linux-gnu/liblz4.so.1.8.3 ceph-osd 225860 ceph mem REG 9,127 31104 133334 /usr/lib/x86_64-linux-gnu/libsnappy.so.1.1.7 ceph-osd 225860 ceph mem REG 9,127 378160 133086 /usr/lib/x86_64-linux-gnu/libleveldb.so.1d.20 ceph-osd 225860 ceph mem REG 9,127 256120 7650 /lib/x86_64-linux-gnu/libfuse.so.2.9.9 ceph-osd 225860 ceph mem REG 9,127 343008 16890 /lib/x86_64-linux-gnu/libblkid.so.1.1.0 ceph-osd 225860 ceph mem REG 8,70 1546 68022625 /var/lib/ceph/osd/ceph-0/current/omap/165542.ldb ceph-osd 225860 ceph mem REG 8,70 1598 67109142 /var/lib/ceph/osd/ceph-0/current/omap/165533.ldb ceph-osd 225860 ceph mem REG 9,127 35112 133884 /usr/lib/x86_64-linux-gnu/liburcu-bp.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 165632 15000 /lib/x86_64-linux-gnu/ld-2.28.so ceph-osd 225860 ceph 0r CHR 1,3 0t0 6 /dev/null ceph-osd 225860 ceph 1u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 2u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 3u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 4r FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 5w FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 6u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 7r FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 8w FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 9u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 10r FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 11w FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 12w REG 9,127 8765 3314 /var/log/ceph/ceph-osd.0.log ceph-osd 225860 ceph 13r FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 14w FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 15u unix 0x0000000097fdc46a 0t0 64259951 /var/run/ceph/ceph-osd.0.asok type=STREAM ceph-osd 225860 ceph 24r FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 25w FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 26r FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 27w FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 28uW REG 8,70 37 21 /var/lib/ceph/osd/ceph-0/fsid ceph-osd 225860 ceph 29r DIR 8,70 262 16 /var/lib/ceph/osd/ceph-0 ceph-osd 225860 ceph 30r DIR 8,70 24576 27 /var/lib/ceph/osd/ceph-0/current ceph-osd 225860 ceph 31u REG 8,70 10 28 /var/lib/ceph/osd/ceph-0/current/commit_op_seq ceph-osd 225860 ceph 32uW REG 8,70 0 67108882 /var/lib/ceph/osd/ceph-0/current/omap/LOCK ceph-osd 225860 ceph 33w REG 8,70 813568 68035819 /var/lib/ceph/osd/ceph-0/current/omap/165540.log ceph-osd 225860 ceph 34w REG 8,70 32068 68072533 /var/lib/ceph/osd/ceph-0/current/omap/MANIFEST-165538 ceph-osd 225860 ceph 36u BLK 8,65 0t0 422 /dev/sde1 ceph-osd 225860 ceph 37u REG 8,70 538 134217745 /var/lib/ceph/osd/ceph-0/current/meta/DIR_E/DIR_D/DIR_C/osd\\usuperblock__0_23C2FCDE__none ceph-osd 225860 ceph 38u REG 8,70 22453 1075330317 /var/lib/ceph/osd/ceph-0/current/meta/DIR_4/DIR_B/DIR_4/osdmap.126556__0_DF0524B4__none strace shows strace: Process 225860 attached futex(0x555f3e1c05c8, FUTEX_WAIT_PRIVATE, 0, NULL -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 https://www.imatic.cz | +420326555326 ============ --

3 8

CompleteMultipartUploadResult has empty ETag response
by Lars Dunemark 28 Feb '23

28 Feb '23

Hi, I notice that CompleteMultipartUploadResult does return an empty ETag field when completing an multipart upload in v17.2.3. I haven't had the possibility to verify from which version this changed and can't find in the changelog that it should be fixed in newer version. The response looks like: <?xml version="1.0" encoding="UTF-8"?> <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/ <http://s3.amazonaws.com/doc/2006-03-01/>"> <Location>s3.myceph.com/test-bucket/test.file</Location> <Bucket>test-bucket</Bucket> <Key>test.file</Key> <ETag></ETag> </CompleteMultipartUploadResult> I have found a old issue that is closed around 9 years ago with the same issue so I guess that this has been fixed before. https://tracker.ceph.com/issues/6830 <https://tracker.ceph.com/issues/6830> It looks like my account to the tracker is still not activated so I can't create or comment on the issue. Best regards, Lars Dunemark

2 1

Upgrade cephadm cluster
by Nicola Mori 28 Feb '23

28 Feb '23

Dear Ceph users, my cephadm-managed cluster is currently based on 17.2.3. I see that 17.2.5 is available on quay.io, so I'd like to upgrade. I read the upgrade guide (https://docs.ceph.com/en/quincy/cephadm/upgrade/) and the "Potential problems" section is reassuringly short. Still I'm worried about potential problems that might arise during this upgrade: my cluster is in production and I fear about possible data losses or extended downtimes due to upgrade problems. So what should I expect from the upgrade process? Is it usually a smooth experience or are there frequent/known/probable issues? Are there some good practices to minimize the occurrency of problems and to eventually prepare for recovery procedures? Thank you, Nicola

2 2

Upgrade not doing anything...
by Jeremy Hansen 28 Feb '23

28 Feb '23

I’m trying to upgrade from 16.2.7 to 16.2.11. Reading the documentation, I cut and paste the orchestrator command to begin the upgrade, but I mistakenly pasted directly from the docs and it initiated an “upgrade” to 16.2.6. I stopped the upgrade per the docs and reissued the command specifying 16.2.11 but now I see no progress in ceph -s. Cluster is healthy but it feels like the upgrade process is just paused for some reason. Thanks! -jeremy

2 5

mons excessive writes to local disk and SSD wearout
by Andrej Filipcic 27 Feb '23

27 Feb '23

Hi, on our large ceph cluster with 60 servers, 1600 OSDs, we have observed that small system nvmes are wearing out rapidly. Our monitoring shows mon writes on average about 10MB/s to store.db. For small system nvmes of 250GB and DWPD of ~1, this turns out to be too much, 0.8TB/day or 1.5PB in 5 years, too much even for 3DWPD of the same capacity. Apart from replacing the drives with larger ones, more durable, preferably both, do you have any suggestions if these writes can be reduced? Actually, the mon writes match 0.15Hz rate of .sst file creation of 64MB.... Best regards, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

3 3

rbd map error: couldn't connect to the cluster!
by Thomas Schneider 27 Feb '23

27 Feb '23

Hello, I'm trying to mount RBD using rbd map, but I get this error message: # rbd map hdb_backup/VCT --id client --keyring /etc/ceph/ceph.client.VCT.keyring rbd: couldn't connect to the cluster! Checking on Ceph server the required permission for relevant keyring exists: # ceph-authtool -l /etc/ceph/ceph.client.VCT.keyring [client.VCT] key = AQBj3LZjNGn/BhAAG8IqMyH0WLKi4kTlbjiW7g== # ceph auth get client.VCT [client.VCT] key = AQBj3LZjNGn/BhAAG8IqMyH0WLKi4kTlbjiW7g== caps mon = "allow r" caps osd = "allow rwx pool hdb_backup object_prefix rbd_data.b768d4baac048b; allow rwx pool hdb_backup object_prefix rbd_header.b768d4baac048b; allow rx pool hdb_backup object_prefix rbd_id.VCT" exported keyring for client.VCT Can you please advise how to fix this error? THX

4 20

CompleteMultipartUploadResult has empty ETag response
by Lars Dunemark 27 Feb '23

27 Feb '23

Hi, When I was looking at a complete multipart upload request I found that the response did return an empty ETag entry in the XML. If I query the keys metadata after the Complete is done it will return the expected ETag so it looks like it is calculated correctly. <?xml version="1.0" encoding="UTF-8"?> <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Location>testsystem/test/upload-file</Location> <Bucket>test</Bucket> <Key>upload-file</Key> <ETag></ETag> </CompleteMultipartUploadResult> I only have a cluster running on v17.2.3 so I haven't verified if this still exist on latest version. I found an really old issue that was closed ~9 years ago with the same issue. https://tracker.ceph.com/issues/6830 The problem is that my account to the tracker doesn't seams to work as is should so I can't login and comment on it or create a new ticket. It also looks like there is a inconsistency between https://docs.ceph.com/en/latest/radosgw/s3/objectops/#complete-multipart-up… that says that ETag is required in the request but for eg. AWS documentation doesn't have it as a possible argument for the CompleteMultipartUpload so it is not possible to send using common third party libraries. Best regards, Lars Dunemark

1 0

Slow replication of large buckets (after reshard)
by Glaza 27 Feb '23

27 Feb '23

Hi Cephers, We have two octopus 15.2.17 clusters in a multisite configuration. Every once in a while we have to perform a bucket reshard (most recently on 613 shards) and this practically kills our replication for a few days. Does anyone know of any priority mechanics within sync to give priority to other buckets and/or lower them? Are there any improvements to this in higher versions of ceph that we could take advantage of if we upgrade the cluster (I haven't found any)? How to safely perform the increase of rgw_data_log_num_shards, because the documentation only says: "The values of rgw_data_log_num_shards and rgw_md_log_max_shards should not be changed after sync has started." Does this mean that I should block access to the cluster, wait until sync is caught up with source/master, change this value, restart rgw and unblock access? Kind Regards, Tom

1 0

OpenSSL in librados
by Patrick Schlangen 26 Feb '23

26 Feb '23

Hi, please forgive me if this has been asked before - I could not find any information on this topic. I am using ceph with librados via the phprados extension. Since upgrading to the current ceph versions where OpenSSL is used in in librados, I observe that using PHP's libcurl integration and other features which rely on OpenSSL randomly fail when opening a TLS connection. I suspect that librados somehow initializes or uninitializes OpenSSL in a way that interferes with the OpenSSL usage of libcurl / PHP's fsockopen. Did anybody make a similar experience? Thanks, Patrick

3 5

How to see bucket usage when user is suspended ?
by Arnaud M 26 Feb '23

26 Feb '23

Hello to everyone When I use this command to see bucket usage radosgw-admin bucket stats --bucket=<bucket> It work only when the owner of the bucket is activated How to see the usage even when the owner is suspended ? Here is 2 exemple, one with the owner activated et the other one with owner suspended: radosgw-admin bucket stats --bucket=bonjour { "bucket": "bonjour", "num_shards": 11, "tenant": "", "zonegroup": "46d4ba06-76ff-44b4-a441-54197517ded2", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "f8c2e3e2-da22-4c80-b330-466db13bbf6a.204114.85", "marker": "f8c2e3e2-da22-4c80-b330-466db13bbf6a.204114.85", "index_type": "Normal", "owner": "identifiant_leviia_GB6mSIAmTt48cY5O", "ver": "0#148,1#124,2#134,3#155,4#199,5#123,6#165,7#141,8#133,9#154,10#137", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "0.000000", "creation_time": "2023-02-24T16:16:14.196314Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 532572233, "size_actual": 535318528, "size_utilized": 532572233, "size_kb": 520091, "size_kb_actual": 522772, "size_kb_utilized": 520091, "num_objects": 1486 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } radosgw-admin bucket stats --bucket=locking4 { "bucket": "locking4", "num_shards": 11, "tenant": "", "zonegroup": "46d4ba06-76ff-44b4-a441-54197517ded2", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "f8c2e3e2-da22-4c80-b330-466db13bbf6a.204114.80", "marker": "f8c2e3e2-da22-4c80-b330-466db13bbf6a.204114.80", "index_type": "Normal", "owner": "identifiant_leviia_xf4q139fq1", "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "0.000000", "creation_time": "2023-02-23T12:49:24.089538Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": {}, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } As you can see with the bucket where the owner is suspended (locking4) it lack the part: "usage": { "rgw.main": { "size": 532572233, "size_actual": 535318528, "size_utilized": 532572233, "size_kb": 520091, "size_kb_actual": 522772, "size_kb_utilized": 520091, "num_objects": 1486 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }, How to have this part even when the owner is suspended ? Is it possible via API ? All the best

1 0

tools to debug librbd / qemu
by Arvid Picciani 26 Feb '23

26 Feb '23

Heya, ever since we had that one osd causing the entire cluster to hang (it's been removed since), we keep having hard to debug issues. for example sometimes on start, qemu just hangs forever. when i kill it manually, the next start works fine. when i map the same volume using krbd on another host, it also appears fine. How would you debug an issue like this? Since it happens before the vm even starts, i'm assuming it must be the mon that hangs? 'rbd lock ls' is empty, thats not it. is this possibly caused by running inconsistent mon versions? How would i know? ceph tell mon.\* version mon.mon-yca4ceph: { "version": "16.2.11", "release": "pacific", "release_type": "stable" } mon.mon-yca5ceph: { "version": "16.2.6", "release": "pacific", "release_type": "stable" } mon.mon-yca6ceph: { "version": "16.2.11", "release": "pacific", "release_type": "stable" } -- +4916093821054

1 0

Any experience dealing with CephMgrPrometheusModuleInactive?
by Joshua Katz 26 Feb '23

26 Feb '23

Hey all! I'm a first time ceph user trying to learn how to set up a cluster. I've gotten a basic cluster created using the following: ``` cehphadm bootstrap --mon-ip <SERVER_1_IP> ceph orch host add server-2 <SERVER_2_IP> _admin ``` I've created and mounted an fs on a host, everything is going well, but I have noticed that I have an alert triggered: CephMgrPrometheusModuleInactive. It seems this alert is trying to `curl server-2:9283`. To debug if this was a network issue I did `ceph mgr fail` to move the mgr to server-2. After some time I get the same alert with the instance being server-1:9283. Running `ss -l -n -p | grep 9283` shows the port is bound on server-2 and not server-1. If I run `ceph mgr fail` again the port becomes bound on server-1 and not server-2. Is this alert important? Is there a way to remediate this issue? Let me know if I am missing something here. Thanks, - Josh

1 0

Accessing OSD objects
by Geoffrey Rhodes 24 Feb '23

24 Feb '23

Hi Anthony, thanks for reaching out. Erasure data pool (K=4, M=2) but I had more than two disk failures around the same time and the data had not fully replicated / restored elsewhere in the cluster. They are big 12TB Exos so it usually takes a few weeks to backfill / recover plus I had snaptrimming on the go. FYI - The journal's co-located on drive. Kind regards Geoff On Fri, 24 Feb 2023 at 18:30, Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Are you only doing 2 replicas? > > > > > > On Feb 24, 2023, at 08:20, Geoffrey Rhodes <geoffrey(a)rhodes.org.za> wrote: > > This has caused a PG to go inactive > > >

1 0

Large STDDEV in pg per osd
by Joe Ryner 24 Feb '23

24 Feb '23

I have been digging for a while on how to minimize STDDEV of the distribution of data on my OSDs and I can't seem to get it to get below 12. I have other clusters that have a STDDEV of 1 which is my goal. But this cluster is really giving me fits. This cluster started off on the Emperor. Might have even ran Dumpling originally. I'm not sure I have some crud laying around that is preventing better balancing. I have tried the ceph balancer and I even tried jj balancer. As you can see below My lowest osd percent used is 37% and highest is 64% If anyone has any ideas on how to crack this nut I would greatly appreciate it. Thanks, Joe Here is my data: ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 167.63680 root default -3 57.62895 rack marack -4 10.47656 host gallo 0 ssd 3.49219 osd.0 up 1.00000 1.00000 1 ssd 3.49219 osd.1 up 1.00000 1.00000 8 ssd 3.49219 osd.8 up 1.00000 1.00000 -8 7.85712 host jeep 18 ssd 0.87299 osd.18 up 1.00000 1.00000 19 ssd 0.87299 osd.19 up 1.00000 1.00000 20 ssd 0.87299 osd.20 up 1.00000 1.00000 21 ssd 0.87299 osd.21 up 1.00000 1.00000 22 ssd 3.49219 osd.22 up 1.00000 1.00000 45 ssd 0.87299 osd.45 up 1.00000 1.00000 -5 7.85712 host joy 5 ssd 0.87299 osd.5 up 1.00000 1.00000 12 ssd 0.87299 osd.12 up 1.00000 1.00000 13 ssd 0.87299 osd.13 up 1.00000 1.00000 16 ssd 0.87299 osd.16 up 1.00000 1.00000 17 ssd 3.49219 osd.17 up 1.00000 1.00000 43 ssd 0.87299 osd.43 up 1.00000 1.00000 -37 6.98625 host kc01 82 ssd 0.87329 osd.82 up 1.00000 1.00000 83 ssd 0.87329 osd.83 up 1.00000 1.00000 84 ssd 0.87329 osd.84 up 1.00000 1.00000 85 ssd 3.49309 osd.85 up 1.00000 1.00000 86 ssd 0.87329 osd.86 up 1.00000 1.00000 -50 6.98625 host ks02 97 ssd 0.87329 osd.97 up 1.00000 1.00000 98 ssd 0.87329 osd.98 up 1.00000 1.00000 99 ssd 0.87329 osd.99 up 1.00000 1.00000 100 ssd 0.87329 osd.100 up 1.00000 1.00000 101 ssd 3.49309 osd.101 up 1.00000 1.00000 -27 3.49319 host lc02 52 ssd 1.74660 osd.52 up 1.00000 1.00000 73 ssd 1.74660 osd.73 up 1.00000 1.00000 -55 13.97246 host lx01 14 ssd 3.49309 osd.14 up 1.00000 1.00000 27 ssd 3.49309 osd.27 up 1.00000 1.00000 29 ssd 3.49309 osd.29 up 1.00000 1.00000 48 ssd 1.74660 osd.48 up 1.00000 1.00000 49 ssd 1.74660 osd.49 up 1.00000 1.00000 -2 55.00392 rack marack2 -13 13.96875 host helm 11 ssd 3.49219 osd.11 up 1.00000 1.00000 28 ssd 3.49219 osd.28 up 1.00000 1.00000 30 ssd 3.49219 osd.30 up 1.00000 1.00000 31 ssd 3.49219 osd.31 up 1.00000 1.00000 -17 7.85712 host jazz 23 ssd 0.87299 osd.23 up 1.00000 1.00000 24 ssd 0.87299 osd.24 up 1.00000 1.00000 25 ssd 0.87299 osd.25 up 1.00000 1.00000 26 ssd 0.87299 osd.26 up 1.00000 1.00000 36 ssd 3.49219 osd.36 up 1.00000 1.00000 44 ssd 0.87299 osd.44 up 1.00000 1.00000 -6 6.98438 host john 2 ssd 3.49219 osd.2 up 1.00000 1.00000 3 ssd 3.49219 osd.3 up 1.00000 1.00000 -49 7.85712 host jolt 34 ssd 3.49219 osd.34 up 1.00000 1.00000 63 ssd 0.87299 osd.63 up 1.00000 1.00000 64 ssd 0.87299 osd.64 up 1.00000 1.00000 65 ssd 0.87299 osd.65 up 1.00000 1.00000 66 ssd 0.87299 osd.66 up 1.00000 1.00000 67 ssd 0.87299 osd.67 up 1.00000 1.00000 -19 7.85712 host juju 37 ssd 0.87299 osd.37 up 1.00000 1.00000 38 ssd 0.87299 osd.38 up 1.00000 1.00000 39 ssd 0.87299 osd.39 up 1.00000 1.00000 40 ssd 0.87299 osd.40 up 1.00000 1.00000 41 ssd 3.49219 osd.41 up 1.00000 1.00000 46 ssd 0.87299 osd.46 up 1.00000 1.00000 -44 6.98625 host ks01 92 ssd 0.87329 osd.92 up 1.00000 1.00000 93 ssd 0.87329 osd.93 up 1.00000 1.00000 94 ssd 0.87329 osd.94 up 1.00000 1.00000 95 ssd 0.87329 osd.95 up 1.00000 1.00000 96 ssd 3.49309 osd.96 up 1.00000 1.00000 -22 3.49319 host lc01 50 ssd 1.74660 osd.50 up 1.00000 1.00000 51 ssd 1.74660 osd.51 up 1.00000 1.00000 -11 55.00392 rack marack3 -10 13.96875 host gold 7 ssd 3.49219 osd.7 up 1.00000 1.00000 9 ssd 3.49219 osd.9 up 1.00000 1.00000 10 ssd 3.49219 osd.10 up 1.00000 1.00000 15 ssd 3.49219 osd.15 up 1.00000 1.00000 -12 3.49319 host innes 42 ssd 1.74660 osd.42 up 1.00000 1.00000 47 ssd 1.74660 osd.47 up 1.00000 1.00000 -16 6.98438 host jack 4 ssd 3.49219 osd.4 up 1.00000 1.00000 6 ssd 3.49219 osd.6 up 1.00000 1.00000 -43 7.85712 host jam 32 ssd 3.49219 osd.32 up 1.00000 1.00000 53 ssd 0.87299 osd.53 up 1.00000 1.00000 54 ssd 0.87299 osd.54 up 1.00000 1.00000 55 ssd 0.87299 osd.55 up 1.00000 1.00000 56 ssd 0.87299 osd.56 up 1.00000 1.00000 57 ssd 0.87299 osd.57 up 1.00000 1.00000 -46 7.85712 host jet 33 ssd 3.49219 osd.33 up 1.00000 1.00000 58 ssd 0.87299 osd.58 up 1.00000 1.00000 59 ssd 0.87299 osd.59 up 1.00000 1.00000 60 ssd 0.87299 osd.60 up 1.00000 1.00000 61 ssd 0.87299 osd.61 up 1.00000 1.00000 62 ssd 0.87299 osd.62 up 1.00000 1.00000 -52 7.85712 host jug 35 ssd 3.49219 osd.35 up 1.00000 1.00000 68 ssd 0.87299 osd.68 up 1.00000 1.00000 69 ssd 0.87299 osd.69 up 1.00000 1.00000 70 ssd 0.87299 osd.70 up 1.00000 1.00000 71 ssd 0.87299 osd.71 up 1.00000 1.00000 72 ssd 0.87299 osd.72 up 1.00000 1.00000 -41 6.98625 host kc02 87 ssd 0.87329 osd.87 up 1.00000 1.00000 88 ssd 0.87329 osd.88 up 1.00000 1.00000 89 ssd 0.87329 osd.89 up 1.00000 1.00000 90 ssd 3.49309 osd.90 up 1.00000 1.00000 91 ssd 0.87329 osd.91 up 1.00000 1.00000 ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 167.63680 - 168 TiB 80 TiB 79 TiB 331 MiB 282 GiB 88 TiB 47.56 1.00 - root default -3 57.62895 - 58 TiB 26 TiB 26 TiB 106 MiB 96 GiB 31 TiB 45.78 0.96 - rack marack -4 10.47656 - 10 TiB 4.3 TiB 4.3 TiB 30 MiB 14 GiB 6.1 TiB 41.46 0.87 - host gallo 0 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.4 TiB 9.9 MiB 4.8 GiB 2.0 TiB 41.53 0.87 141 up osd.0 1 ssd 3.49219 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 3.7 MiB 4.7 GiB 2.1 TiB 40.42 0.85 136 up osd.1 8 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 16 MiB 4.8 GiB 2.0 TiB 42.44 0.89 132 up osd.8 -8 7.85712 - 7.9 TiB 4.1 TiB 4.1 TiB 14 MiB 14 GiB 3.7 TiB 52.30 1.10 - host jeep 18 ssd 0.87299 1.00000 894 GiB 569 GiB 567 GiB 987 KiB 2.0 GiB 325 GiB 63.65 1.34 35 up osd.18 19 ssd 0.87299 1.00000 894 GiB 552 GiB 550 GiB 991 KiB 1.9 GiB 342 GiB 61.72 1.30 32 up osd.19 20 ssd 0.87299 1.00000 894 GiB 549 GiB 548 GiB 945 KiB 1.8 GiB 345 GiB 61.46 1.29 69 up osd.20 21 ssd 0.87299 1.00000 894 GiB 577 GiB 575 GiB 1.0 MiB 1.8 GiB 317 GiB 64.57 1.36 33 up osd.21 22 ssd 3.49219 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 8.8 MiB 4.8 GiB 2.1 TiB 39.21 0.82 133 up osd.22 45 ssd 0.87299 1.00000 894 GiB 558 GiB 556 GiB 938 KiB 1.9 GiB 336 GiB 62.41 1.31 35 up osd.45 -5 7.85712 - 7.9 TiB 4.2 TiB 4.1 TiB 16 MiB 14 GiB 3.7 TiB 52.85 1.11 - host joy 5 ssd 0.87299 1.00000 894 GiB 569 GiB 567 GiB 968 KiB 1.9 GiB 325 GiB 63.60 1.34 32 up osd.5 12 ssd 0.87299 1.00000 894 GiB 576 GiB 575 GiB 1006 KiB 1.9 GiB 318 GiB 64.49 1.36 33 up osd.12 13 ssd 0.87299 1.00000 894 GiB 578 GiB 576 GiB 1.0 MiB 1.9 GiB 316 GiB 64.65 1.36 33 up osd.13 16 ssd 0.87299 1.00000 894 GiB 569 GiB 567 GiB 1015 KiB 1.9 GiB 325 GiB 63.68 1.34 35 up osd.16 17 ssd 3.49219 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 11 MiB 4.6 GiB 2.1 TiB 39.36 0.83 130 up osd.17 43 ssd 0.87299 1.00000 894 GiB 553 GiB 551 GiB 968 KiB 1.9 GiB 341 GiB 61.81 1.30 33 up osd.43 -37 6.98625 - 7.0 TiB 3.3 TiB 3.3 TiB 9.0 MiB 13 GiB 3.7 TiB 46.81 0.98 - host kc01 82 ssd 0.87329 1.00000 894 GiB 497 GiB 495 GiB 984 KiB 1.8 GiB 397 GiB 55.58 1.17 34 up osd.82 83 ssd 0.87329 1.00000 894 GiB 496 GiB 494 GiB 1.6 MiB 1.9 GiB 399 GiB 55.42 1.17 36 up osd.83 84 ssd 0.87329 1.00000 894 GiB 513 GiB 511 GiB 1.0 MiB 1.9 GiB 381 GiB 57.34 1.21 35 up osd.84 85 ssd 3.49309 1.00000 3.5 TiB 1.3 TiB 1.3 TiB 4.5 MiB 5.7 GiB 2.2 TiB 37.68 0.79 140 up osd.85 86 ssd 0.87329 1.00000 894 GiB 495 GiB 493 GiB 931 KiB 1.9 GiB 399 GiB 55.37 1.16 43 up osd.86 -50 6.98625 - 7.0 TiB 3.3 TiB 3.3 TiB 18 MiB 13 GiB 3.7 TiB 47.32 0.99 - host ks02 97 ssd 0.87329 1.00000 894 GiB 493 GiB 491 GiB 934 KiB 1.9 GiB 401 GiB 55.11 1.16 100 up osd.97 98 ssd 0.87329 1.00000 894 GiB 518 GiB 516 GiB 1.5 MiB 1.9 GiB 376 GiB 57.92 1.22 33 up osd.98 99 ssd 0.87329 1.00000 894 GiB 514 GiB 512 GiB 1019 KiB 1.7 GiB 380 GiB 57.46 1.21 38 up osd.99 100 ssd 0.87329 1.00000 894 GiB 514 GiB 512 GiB 1.0 MiB 1.9 GiB 381 GiB 57.44 1.21 35 up osd.100 101 ssd 3.49309 1.00000 3.5 TiB 1.3 TiB 1.3 TiB 13 MiB 5.3 GiB 2.2 TiB 37.65 0.79 140 up osd.101 -27 3.49319 - 3.5 TiB 1.6 TiB 1.6 TiB 4.2 MiB 6.9 GiB 1.9 TiB 45.58 0.96 - host lc02 52 ssd 1.74660 1.00000 1.7 TiB 799 GiB 796 GiB 2.0 MiB 3.4 GiB 989 GiB 44.70 0.94 70 up osd.52 73 ssd 1.74660 1.00000 1.7 TiB 831 GiB 828 GiB 2.2 MiB 3.5 GiB 957 GiB 46.47 0.98 66 up osd.73 -55 13.97246 - 14 TiB 5.6 TiB 5.6 TiB 16 MiB 20 GiB 8.4 TiB 40.15 0.84 - host lx01 14 ssd 3.49309 1.00000 3.5 TiB 1.3 TiB 1.3 TiB 3.8 MiB 3.9 GiB 2.2 TiB 37.83 0.80 140 up osd.14 27 ssd 3.49309 1.00000 3.5 TiB 1.3 TiB 1.3 TiB 3.9 MiB 4.7 GiB 2.2 TiB 37.65 0.79 140 up osd.27 29 ssd 3.49309 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 4.1 MiB 4.8 GiB 2.1 TiB 40.47 0.85 133 up osd.29 48 ssd 1.74660 1.00000 1.7 TiB 817 GiB 813 GiB 2.1 MiB 3.2 GiB 972 GiB 45.66 0.96 71 up osd.48 49 ssd 1.74660 1.00000 1.7 TiB 780 GiB 776 GiB 2.0 MiB 3.5 GiB 1009 GiB 43.61 0.92 69 up osd.49 -2 55.00392 - 55 TiB 27 TiB 27 TiB 118 MiB 94 GiB 28 TiB 48.50 1.02 - rack marack2 -13 13.96875 - 14 TiB 5.9 TiB 5.9 TiB 20 MiB 20 GiB 8.0 TiB 42.41 0.89 - host helm 11 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 3.9 MiB 4.8 GiB 2.0 TiB 42.34 0.89 132 up osd.11 28 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 7.3 MiB 5.0 GiB 2.0 TiB 42.48 0.89 132 up osd.28 30 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 4.1 MiB 5.1 GiB 2.0 TiB 42.44 0.89 132 up osd.30 31 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 4.7 MiB 5.1 GiB 2.0 TiB 42.40 0.89 132 up osd.31 -17 7.85712 - 7.9 TiB 4.3 TiB 4.3 TiB 20 MiB 15 GiB 3.6 TiB 54.33 1.14 - host jazz 23 ssd 0.87299 1.00000 894 GiB 570 GiB 568 GiB 1003 KiB 1.9 GiB 324 GiB 63.81 1.34 51 up osd.23 24 ssd 0.87299 1.00000 894 GiB 572 GiB 570 GiB 1.0 MiB 2.0 GiB 322 GiB 63.99 1.35 34 up osd.24 25 ssd 0.87299 1.00000 894 GiB 576 GiB 574 GiB 1.0 MiB 1.9 GiB 318 GiB 64.39 1.35 33 up osd.25 26 ssd 0.87299 1.00000 894 GiB 571 GiB 569 GiB 1018 KiB 2.0 GiB 323 GiB 63.91 1.34 49 up osd.26 36 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 15 MiB 5.0 GiB 2.0 TiB 42.23 0.89 131 up osd.36 44 ssd 0.87299 1.00000 894 GiB 572 GiB 570 GiB 1.0 MiB 1.9 GiB 322 GiB 63.96 1.34 49 up osd.44 -6 6.98438 - 7.0 TiB 2.9 TiB 2.9 TiB 27 MiB 9.9 GiB 4.1 TiB 41.83 0.88 - host john 2 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 16 MiB 4.8 GiB 2.0 TiB 42.33 0.89 132 up osd.2 3 ssd 3.49219 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 11 MiB 5.1 GiB 2.0 TiB 41.33 0.87 141 up osd.3 -49 7.85712 - 7.9 TiB 4.2 TiB 4.2 TiB 17 MiB 14 GiB 3.6 TiB 53.92 1.13 - host jolt 34 ssd 3.49219 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 9.6 MiB 4.8 GiB 2.0 TiB 41.43 0.87 142 up osd.34 63 ssd 0.87299 1.00000 894 GiB 571 GiB 569 GiB 1023 KiB 1.9 GiB 323 GiB 63.88 1.34 45 up osd.63 64 ssd 0.87299 1.00000 894 GiB 571 GiB 569 GiB 1.0 MiB 2.0 GiB 323 GiB 63.91 1.34 34 up osd.64 65 ssd 0.87299 1.00000 894 GiB 571 GiB 569 GiB 983 KiB 1.9 GiB 323 GiB 63.86 1.34 39 up osd.65 66 ssd 0.87299 1.00000 894 GiB 571 GiB 569 GiB 992 KiB 1.9 GiB 323 GiB 63.91 1.34 41 up osd.66 67 ssd 0.87299 1.00000 894 GiB 572 GiB 570 GiB 3.6 MiB 1.9 GiB 322 GiB 64.00 1.35 34 up osd.67 -19 7.85712 - 7.9 TiB 4.3 TiB 4.3 TiB 14 MiB 15 GiB 3.6 TiB 54.34 1.14 - host juju 37 ssd 0.87299 1.00000 894 GiB 574 GiB 572 GiB 1.0 MiB 1.9 GiB 320 GiB 64.23 1.35 33 up osd.37 38 ssd 0.87299 1.00000 894 GiB 570 GiB 569 GiB 1003 KiB 1.8 GiB 324 GiB 63.80 1.34 42 up osd.38 39 ssd 0.87299 1.00000 894 GiB 577 GiB 575 GiB 1.1 MiB 1.7 GiB 317 GiB 64.51 1.36 33 up osd.39 40 ssd 0.87299 1.00000 894 GiB 577 GiB 575 GiB 1004 KiB 1.9 GiB 317 GiB 64.51 1.36 33 up osd.40 41 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 9.2 MiB 5.4 GiB 2.0 TiB 41.97 0.88 135 up osd.41 46 ssd 0.87299 1.00000 894 GiB 573 GiB 571 GiB 1003 KiB 1.8 GiB 321 GiB 64.09 1.35 33 up osd.46 -44 6.98625 - 7.0 TiB 3.4 TiB 3.4 TiB 16 MiB 13 GiB 3.5 TiB 49.37 1.04 - host ks01 92 ssd 0.87329 1.00000 894 GiB 522 GiB 520 GiB 1.0 MiB 2.0 GiB 372 GiB 58.35 1.23 33 up osd.92 93 ssd 0.87329 1.00000 894 GiB 520 GiB 518 GiB 1004 KiB 1.8 GiB 375 GiB 58.10 1.22 33 up osd.93 94 ssd 0.87329 1.00000 894 GiB 515 GiB 513 GiB 1.7 MiB 2.0 GiB 379 GiB 57.61 1.21 48 up osd.94 95 ssd 0.87329 1.00000 894 GiB 515 GiB 514 GiB 1.0 MiB 1.8 GiB 379 GiB 57.63 1.21 35 up osd.95 96 ssd 3.49309 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 11 MiB 5.7 GiB 2.1 TiB 40.83 0.86 132 up osd.96 -22 3.49319 - 3.5 TiB 1.6 TiB 1.6 TiB 4.1 MiB 7.1 GiB 1.9 TiB 45.95 0.97 - host lc01 50 ssd 1.74660 1.00000 1.7 TiB 820 GiB 817 GiB 2.1 MiB 3.5 GiB 968 GiB 45.87 0.96 76 up osd.50 51 ssd 1.74660 1.00000 1.7 TiB 823 GiB 819 GiB 2.0 MiB 3.6 GiB 965 GiB 46.02 0.97 78 up osd.51 -11 55.00392 - 55 TiB 27 TiB 27 TiB 106 MiB 92 GiB 28 TiB 48.49 1.02 - rack marack3 -10 13.96875 - 14 TiB 5.9 TiB 5.8 TiB 32 MiB 20 GiB 8.1 TiB 41.90 0.88 - host gold 7 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 4.0 MiB 4.8 GiB 2.0 TiB 42.24 0.89 131 up osd.7 9 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.4 TiB 9.3 MiB 4.9 GiB 2.0 TiB 41.61 0.87 140 up osd.9 10 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 4.0 MiB 5.0 GiB 2.0 TiB 42.20 0.89 131 up osd.10 15 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.4 TiB 14 MiB 4.8 GiB 2.0 TiB 41.54 0.87 143 up osd.15 -12 3.49319 - 3.5 TiB 1.6 TiB 1.6 TiB 4.2 MiB 6.6 GiB 1.9 TiB 46.48 0.98 - host innes 42 ssd 1.74660 1.00000 1.7 TiB 834 GiB 830 GiB 2.1 MiB 3.2 GiB 955 GiB 46.61 0.98 66 up osd.42 47 ssd 1.74660 1.00000 1.7 TiB 829 GiB 826 GiB 2.1 MiB 3.4 GiB 959 GiB 46.36 0.97 67 up osd.47 -16 6.98438 - 7.0 TiB 3.0 TiB 2.9 TiB 15 MiB 10 GiB 4.0 TiB 42.30 0.89 - host jack 4 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 11 MiB 5.1 GiB 2.0 TiB 42.31 0.89 131 up osd.4 6 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 4.1 MiB 4.9 GiB 2.0 TiB 42.30 0.89 132 up osd.6 -43 7.85712 - 7.9 TiB 4.3 TiB 4.3 TiB 14 MiB 14 GiB 3.6 TiB 54.53 1.15 - host jam 32 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.5 TiB 9.1 MiB 5.0 GiB 2.0 TiB 42.58 0.90 133 up osd.32 53 ssd 0.87299 1.00000 894 GiB 574 GiB 572 GiB 1011 KiB 1.9 GiB 320 GiB 64.17 1.35 34 up osd.53 54 ssd 0.87299 1.00000 894 GiB 574 GiB 572 GiB 1.0 MiB 2.0 GiB 320 GiB 64.19 1.35 33 up osd.54 55 ssd 0.87299 1.00000 894 GiB 572 GiB 571 GiB 984 KiB 1.8 GiB 322 GiB 64.02 1.35 48 up osd.55 56 ssd 0.87299 1.00000 894 GiB 572 GiB 570 GiB 1003 KiB 1.9 GiB 322 GiB 64.03 1.35 103 up osd.56 57 ssd 0.87299 1.00000 894 GiB 573 GiB 571 GiB 995 KiB 1.7 GiB 321 GiB 64.06 1.35 35 up osd.57 -46 7.85712 - 7.9 TiB 4.3 TiB 4.2 TiB 19 MiB 14 GiB 3.6 TiB 54.20 1.14 - host jet 33 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.4 TiB 13 MiB 5.0 GiB 2.0 TiB 41.61 0.87 142 up osd.33 58 ssd 0.87299 1.00000 894 GiB 573 GiB 571 GiB 1004 KiB 2.0 GiB 321 GiB 64.05 1.35 35 up osd.58 59 ssd 0.87299 1.00000 894 GiB 577 GiB 575 GiB 1.3 MiB 1.8 GiB 317 GiB 64.55 1.36 33 up osd.59 60 ssd 0.87299 1.00000 894 GiB 573 GiB 571 GiB 1019 KiB 1.9 GiB 321 GiB 64.11 1.35 34 up osd.60 61 ssd 0.87299 1.00000 894 GiB 575 GiB 573 GiB 1011 KiB 1.9 GiB 319 GiB 64.29 1.35 33 up osd.61 62 ssd 0.87299 1.00000 894 GiB 575 GiB 573 GiB 1.0 MiB 1.7 GiB 319 GiB 64.31 1.35 33 up osd.62 -52 7.85712 - 7.9 TiB 4.3 TiB 4.2 TiB 13 MiB 14 GiB 3.6 TiB 54.23 1.14 - host jug 35 ssd 3.49219 1.00000 3.5 TiB 1.5 TiB 1.4 TiB 8.2 MiB 4.9 GiB 2.0 TiB 41.64 0.88 142 up osd.35 68 ssd 0.87299 1.00000 894 GiB 576 GiB 574 GiB 1.0 MiB 1.9 GiB 318 GiB 64.42 1.35 33 up osd.68 69 ssd 0.87299 1.00000 894 GiB 574 GiB 572 GiB 1000 KiB 2.0 GiB 320 GiB 64.17 1.35 34 up osd.69 70 ssd 0.87299 1.00000 894 GiB 573 GiB 571 GiB 1011 KiB 1.9 GiB 321 GiB 64.07 1.35 35 up osd.70 71 ssd 0.87299 1.00000 894 GiB 574 GiB 572 GiB 988 KiB 1.9 GiB 320 GiB 64.15 1.35 33 up osd.71 72 ssd 0.87299 1.00000 894 GiB 578 GiB 576 GiB 1.0 MiB 2.0 GiB 316 GiB 64.67 1.36 33 up osd.72 -41 6.98625 - 7.0 TiB 3.4 TiB 3.4 TiB 9.4 MiB 13 GiB 3.5 TiB 49.23 1.04 - host kc02 87 ssd 0.87329 1.00000 894 GiB 521 GiB 519 GiB 1.0 MiB 1.9 GiB 374 GiB 58.23 1.22 33 up osd.87 88 ssd 0.87329 1.00000 894 GiB 521 GiB 519 GiB 1012 KiB 1.8 GiB 373 GiB 58.29 1.23 33 up osd.88 89 ssd 0.87329 1.00000 894 GiB 519 GiB 517 GiB 1.4 MiB 1.9 GiB 376 GiB 57.99 1.22 33 up osd.89 90 ssd 3.49309 1.00000 3.5 TiB 1.4 TiB 1.4 TiB 5.0 MiB 5.2 GiB 2.1 TiB 40.31 0.85 148 up osd.90 91 ssd 0.87329 1.00000 894 GiB 519 GiB 517 GiB 997 KiB 2.0 GiB 375 GiB 58.07 1.22 33 up osd.91 TOTAL 168 TiB 80 TiB 79 TiB 331 MiB 282 GiB 88 TiB 47.56 MIN/MAX VAR: 0.79/1.36 STDDEV: 12.03 placementoptimizer.py show --osds --sort-utilization poolid name type size min pg_num stored used avail shrdsize crush 0 data repl 3 1 32 118.5M 355.9M 17.0T 3.7M 0:data default*1.000 1 metadata repl 3 1 32 13.2K 1.1M 17.0T 423.9B 1:metadata default*1.000 2 rbd repl 3 1 32 0.0B 0.0B 17.0T 0.0B 2:rbd default*1.000 3 images repl 3 1 2048 19.0T 56.9T 17.0T 9.5G 0:data default*1.000 4 locks repl 3 1 32 595.2M 1.8G 17.0T 18.6M 0:data default*1.000 24 device_health_metrics repl 3 2 1 6.7M 20.0M 17.0T 6.7M 0:data default*1.000 25 k8s-101v repl 3 1 32 113.3M 341.3M 17.0T 3.5M 0:data default*1.000 26 k8s-102pm repl 3 2 32 50.4G 151.6G 17.0T 1.6G 0:data default*1.000 27 k8s-103pm repl 3 2 32 25.0B 64.0K 17.0T 0.8B 0:data default*1.000 default 19.02T 57.06T sum 19.02T 57.06T osdid hostname cls devsize weight cweight util pg_num pools 27 lx01 ssd 3.5T 1.00 3.5T 37.6% 140 0 1 2 3 4 25 26 27 101 ks02 ssd 3.5T 1.00 3.5T 37.7% 140 0 1 2 3 4 25 26 27 85 kc01 ssd 3.5T 1.00 3.5T 37.7% 140 0 1 2 3 4 25 26 27 14 lx01 ssd 3.5T 1.00 3.5T 37.8% 140 0 1 2 3 4 25 26 27 22 jeep ssd 3.5T 1.00 3.5T 39.2% 133 0 1 2 3 4 25 26 27 17 joy ssd 3.5T 1.00 3.5T 39.4% 130 0 1 2 3 4 25 26 27 90 kc02 ssd 3.5T 1.00 3.5T 40.3% 148 0 1 2 3 4 25 26 27 1 gallo ssd 3.5T 1.00 3.5T 40.4% 136 0 1 2 3 4 25 26 27 29 lx01 ssd 3.5T 1.00 3.5T 40.5% 133 3 26 27 96 ks01 ssd 3.5T 1.00 3.5T 40.8% 132 3 3 john ssd 3.5T 1.00 3.5T 41.3% 141 0 1 2 3 4 25 26 27 34 jolt ssd 3.5T 1.00 3.5T 41.4% 142 0 1 2 3 4 24 25 26 27 0 gallo ssd 3.5T 1.00 3.5T 41.5% 141 0 1 2 3 4 25 26 27 15 gold ssd 3.5T 1.00 3.5T 41.5% 143 0 1 2 3 4 24 25 26 27 33 jet ssd 3.5T 1.00 3.5T 41.6% 142 0 1 2 3 4 25 26 27 9 gold ssd 3.5T 1.00 3.5T 41.6% 140 0 1 2 3 4 25 26 27 35 jug ssd 3.5T 1.00 3.5T 41.6% 142 0 1 2 3 4 25 26 27 41 juju ssd 3.5T 1.00 3.5T 42.0% 135 0 3 4 25 26 27 10 gold ssd 3.5T 1.00 3.5T 42.2% 131 3 36 jazz ssd 3.5T 1.00 3.5T 42.2% 131 3 7 gold ssd 3.5T 1.00 3.5T 42.2% 131 3 6 jack ssd 3.5T 1.00 3.5T 42.3% 132 3 27 4 jack ssd 3.5T 1.00 3.5T 42.3% 131 3 2 john ssd 3.5T 1.00 3.5T 42.3% 132 3 11 helm ssd 3.5T 1.00 3.5T 42.3% 132 3 31 helm ssd 3.5T 1.00 3.5T 42.4% 132 3 30 helm ssd 3.5T 1.00 3.5T 42.4% 132 3 8 gallo ssd 3.5T 1.00 3.5T 42.4% 132 3 28 helm ssd 3.5T 1.00 3.5T 42.5% 132 3 32 jam ssd 3.5T 1.00 3.5T 42.6% 133 3 49 lx01 ssd 1.7T 1.00 1.7T 43.6% 69 0 1 2 3 4 25 26 27 52 lc02 ssd 1.7T 1.00 1.7T 44.7% 70 0 1 2 3 4 25 26 27 48 lx01 ssd 1.7T 1.00 1.7T 45.6% 71 0 1 2 3 4 25 26 27 50 lc01 ssd 1.7T 1.00 1.7T 45.9% 76 0 1 2 3 4 25 26 27 51 lc01 ssd 1.7T 1.00 1.7T 46.0% 78 0 1 2 3 4 25 26 27 47 innes ssd 1.7T 1.00 1.7T 46.3% 67 3 26 73 lc02 ssd 1.7T 1.00 1.7T 46.5% 66 3 42 innes ssd 1.7T 1.00 1.7T 46.6% 66 3 97 ks02 ssd 894.2G 1.00 894.2G 55.1% 100 0 1 2 3 4 25 26 27 86 kc01 ssd 894.2G 1.00 894.2G 55.4% 43 1 2 3 24 26 83 kc01 ssd 894.2G 1.00 894.2G 55.4% 36 1 2 3 4 25 26 27 82 kc01 ssd 894.2G 1.00 894.2G 55.6% 34 1 3 25 26 27 84 kc01 ssd 894.2G 1.00 894.2G 57.3% 35 0 3 25 26 100 ks02 ssd 894.2G 1.00 894.2G 57.4% 35 1 2 3 27 99 ks02 ssd 894.2G 1.00 894.2G 57.5% 38 0 1 2 3 4 25 26 94 ks01 ssd 894.2G 1.00 894.2G 57.6% 48 0 3 4 25 26 27 95 ks01 ssd 894.2G 1.00 894.2G 57.6% 35 0 3 25 26 98 ks02 ssd 894.2G 1.00 894.2G 57.9% 33 3 89 kc02 ssd 894.2G 1.00 894.2G 58.0% 33 3 91 kc02 ssd 894.2G 1.00 894.2G 58.1% 33 3 93 ks01 ssd 894.2G 1.00 894.2G 58.1% 33 3 87 kc02 ssd 894.2G 1.00 894.2G 58.2% 33 3 88 kc02 ssd 894.2G 1.00 894.2G 58.3% 33 3 92 ks01 ssd 894.2G 1.00 894.2G 58.3% 33 3 20 jeep ssd 894.0G 1.00 893.9G 61.5% 69 1 2 3 4 26 19 jeep ssd 894.0G 1.00 893.9G 61.7% 32 3 25 26 43 joy ssd 894.0G 1.00 893.9G 61.8% 33 0 3 4 26 45 jeep ssd 894.0G 1.00 893.9G 62.4% 35 0 1 2 3 4 5 joy ssd 894.0G 1.00 893.9G 63.6% 32 3 18 jeep ssd 894.0G 1.00 893.9G 63.6% 35 1 3 26 27 16 joy ssd 894.0G 1.00 893.9G 63.7% 35 1 3 26 27 23 jazz ssd 894.0G 1.00 893.9G 63.8% 51 0 3 4 25 26 27 38 juju ssd 894.0G 1.00 893.9G 63.8% 42 0 3 4 25 26 27 65 jolt ssd 894.0G 1.00 893.9G 63.9% 39 0 1 2 3 4 25 26 63 jolt ssd 894.0G 1.00 893.9G 63.9% 45 0 3 4 25 26 27 26 jazz ssd 894.0G 1.00 893.9G 63.9% 49 0 3 4 25 26 27 64 jolt ssd 894.0G 1.00 893.9G 63.9% 34 3 26 27 66 jolt ssd 894.0G 1.00 893.9G 63.9% 41 0 1 2 3 4 25 26 44 jazz ssd 894.0G 1.00 893.9G 63.9% 49 0 3 4 25 26 27 24 jazz ssd 894.0G 1.00 893.9G 64.0% 34 3 26 67 jolt ssd 894.0G 1.00 893.9G 64.0% 34 0 3 26 55 jam ssd 894.0G 1.00 893.9G 64.0% 48 0 3 4 25 26 58 jet ssd 894.0G 1.00 893.9G 64.0% 35 3 26 56 jam ssd 894.0G 1.00 893.9G 64.0% 103 0 3 4 25 26 27 70 jug ssd 894.0G 1.00 893.9G 64.1% 35 3 26 57 jam ssd 894.0G 1.00 893.9G 64.1% 35 3 26 46 juju ssd 894.0G 1.00 893.9G 64.1% 33 3 26 60 jet ssd 894.0G 1.00 893.9G 64.1% 34 3 26 71 jug ssd 894.0G 1.00 893.9G 64.1% 33 3 69 jug ssd 894.0G 1.00 893.9G 64.2% 34 3 26 53 jam ssd 894.0G 1.00 893.9G 64.2% 34 3 26 54 jam ssd 894.0G 1.00 893.9G 64.2% 33 3 37 juju ssd 894.0G 1.00 893.9G 64.2% 33 3 26 61 jet ssd 894.0G 1.00 893.9G 64.3% 33 3 62 jet ssd 894.0G 1.00 893.9G 64.3% 33 3 25 jazz ssd 894.0G 1.00 893.9G 64.4% 33 3 68 jug ssd 894.0G 1.00 893.9G 64.4% 33 3 12 joy ssd 894.0G 1.00 893.9G 64.5% 33 3 40 juju ssd 894.0G 1.00 893.9G 64.5% 33 3 39 juju ssd 894.0G 1.00 893.9G 64.5% 33 3 59 jet ssd 894.0G 1.00 893.9G 64.6% 33 3 21 jeep ssd 894.0G 1.00 893.9G 64.6% 33 3 13 joy ssd 894.0G 1.00 893.9G 64.6% 33 3 72 jug ssd 894.0G 1.00 893.9G 64.7% 33 3 ceph osd crush dump { "devices": [ { "id": 0, "name": "osd.0", "class": "ssd" }, { "id": 1, "name": "osd.1", "class": "ssd" }, { "id": 2, "name": "osd.2", "class": "ssd" }, { "id": 3, "name": "osd.3", "class": "ssd" }, { "id": 4, "name": "osd.4", "class": "ssd" }, { "id": 5, "name": "osd.5", "class": "ssd" }, { "id": 6, "name": "osd.6", "class": "ssd" }, { "id": 7, "name": "osd.7", "class": "ssd" }, { "id": 8, "name": "osd.8", "class": "ssd" }, { "id": 9, "name": "osd.9", "class": "ssd" }, { "id": 10, "name": "osd.10", "class": "ssd" }, { "id": 11, "name": "osd.11", "class": "ssd" }, { "id": 12, "name": "osd.12", "class": "ssd" }, { "id": 13, "name": "osd.13", "class": "ssd" }, { "id": 14, "name": "osd.14", "class": "ssd" }, { "id": 15, "name": "osd.15", "class": "ssd" }, { "id": 16, "name": "osd.16", "class": "ssd" }, { "id": 17, "name": "osd.17", "class": "ssd" }, { "id": 18, "name": "osd.18", "class": "ssd" }, { "id": 19, "name": "osd.19", "class": "ssd" }, { "id": 20, "name": "osd.20", "class": "ssd" }, { "id": 21, "name": "osd.21", "class": "ssd" }, { "id": 22, "name": "osd.22", "class": "ssd" }, { "id": 23, "name": "osd.23", "class": "ssd" }, { "id": 24, "name": "osd.24", "class": "ssd" }, { "id": 25, "name": "osd.25", "class": "ssd" }, { "id": 26, "name": "osd.26", "class": "ssd" }, { "id": 27, "name": "osd.27", "class": "ssd" }, { "id": 28, "name": "osd.28", "class": "ssd" }, { "id": 29, "name": "osd.29", "class": "ssd" }, { "id": 30, "name": "osd.30", "class": "ssd" }, { "id": 31, "name": "osd.31", "class": "ssd" }, { "id": 32, "name": "osd.32", "class": "ssd" }, { "id": 33, "name": "osd.33", "class": "ssd" }, { "id": 34, "name": "osd.34", "class": "ssd" }, { "id": 35, "name": "osd.35", "class": "ssd" }, { "id": 36, "name": "osd.36", "class": "ssd" }, { "id": 37, "name": "osd.37", "class": "ssd" }, { "id": 38, "name": "osd.38", "class": "ssd" }, { "id": 39, "name": "osd.39", "class": "ssd" }, { "id": 40, "name": "osd.40", "class": "ssd" }, { "id": 41, "name": "osd.41", "class": "ssd" }, { "id": 42, "name": "osd.42", "class": "ssd" }, { "id": 43, "name": "osd.43", "class": "ssd" }, { "id": 44, "name": "osd.44", "class": "ssd" }, { "id": 45, "name": "osd.45", "class": "ssd" }, { "id": 46, "name": "osd.46", "class": "ssd" }, { "id": 47, "name": "osd.47", "class": "ssd" }, { "id": 48, "name": "osd.48", "class": "ssd" }, { "id": 49, "name": "osd.49", "class": "ssd" }, { "id": 50, "name": "osd.50", "class": "ssd" }, { "id": 51, "name": "osd.51", "class": "ssd" }, { "id": 52, "name": "osd.52", "class": "ssd" }, { "id": 53, "name": "osd.53", "class": "ssd" }, { "id": 54, "name": "osd.54", "class": "ssd" }, { "id": 55, "name": "osd.55", "class": "ssd" }, { "id": 56, "name": "osd.56", "class": "ssd" }, { "id": 57, "name": "osd.57", "class": "ssd" }, { "id": 58, "name": "osd.58", "class": "ssd" }, { "id": 59, "name": "osd.59", "class": "ssd" }, { "id": 60, "name": "osd.60", "class": "ssd" }, { "id": 61, "name": "osd.61", "class": "ssd" }, { "id": 62, "name": "osd.62", "class": "ssd" }, { "id": 63, "name": "osd.63", "class": "ssd" }, { "id": 64, "name": "osd.64", "class": "ssd" }, { "id": 65, "name": "osd.65", "class": "ssd" }, { "id": 66, "name": "osd.66", "class": "ssd" }, { "id": 67, "name": "osd.67", "class": "ssd" }, { "id": 68, "name": "osd.68", "class": "ssd" }, { "id": 69, "name": "osd.69", "class": "ssd" }, { "id": 70, "name": "osd.70", "class": "ssd" }, { "id": 71, "name": "osd.71", "class": "ssd" }, { "id": 72, "name": "osd.72", "class": "ssd" }, { "id": 73, "name": "osd.73", "class": "ssd" }, { "id": 74, "name": "device74" }, { "id": 75, "name": "device75" }, { "id": 76, "name": "device76" }, { "id": 77, "name": "device77" }, { "id": 78, "name": "device78" }, { "id": 79, "name": "device79" }, { "id": 80, "name": "device80" }, { "id": 81, "name": "device81" }, { "id": 82, "name": "osd.82", "class": "ssd" }, { "id": 83, "name": "osd.83", "class": "ssd" }, { "id": 84, "name": "osd.84", "class": "ssd" }, { "id": 85, "name": "osd.85", "class": "ssd" }, { "id": 86, "name": "osd.86", "class": "ssd" }, { "id": 87, "name": "osd.87", "class": "ssd" }, { "id": 88, "name": "osd.88", "class": "ssd" }, { "id": 89, "name": "osd.89", "class": "ssd" }, { "id": 90, "name": "osd.90", "class": "ssd" }, { "id": 91, "name": "osd.91", "class": "ssd" }, { "id": 92, "name": "osd.92", "class": "ssd" }, { "id": 93, "name": "osd.93", "class": "ssd" }, { "id": 94, "name": "osd.94", "class": "ssd" }, { "id": 95, "name": "osd.95", "class": "ssd" }, { "id": 96, "name": "osd.96", "class": "ssd" }, { "id": 97, "name": "osd.97", "class": "ssd" }, { "id": 98, "name": "osd.98", "class": "ssd" }, { "id": 99, "name": "osd.99", "class": "ssd" }, { "id": 100, "name": "osd.100", "class": "ssd" }, { "id": 101, "name": "osd.101", "class": "ssd" } ], "types": [ { "type_id": 0, "name": "osd" }, { "type_id": 1, "name": "host" }, { "type_id": 2, "name": "rack" }, { "type_id": 3, "name": "row" }, { "type_id": 4, "name": "room" }, { "type_id": 5, "name": "datacenter" }, { "type_id": 6, "name": "root" } ], "buckets": [ { "id": -1, "name": "default", "type_id": 6, "type_name": "root", "weight": 10986245, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -3, "weight": 3776771, "pos": 0 }, { "id": -2, "weight": 3604737, "pos": 1 }, { "id": -11, "weight": 3604737, "pos": 2 } ] }, { "id": -2, "name": "marack2", "type_id": 2, "type_name": "rack", "weight": 3604737, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -13, "weight": 915456, "pos": 0 }, { "id": -19, "weight": 514924, "pos": 1 }, { "id": -49, "weight": 514924, "pos": 2 }, { "id": -17, "weight": 514924, "pos": 3 }, { "id": -44, "weight": 457851, "pos": 4 }, { "id": -6, "weight": 457728, "pos": 5 }, { "id": -22, "weight": 228930, "pos": 6 } ] }, { "id": -3, "name": "marack", "type_id": 2, "type_name": "rack", "weight": 3776771, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -4, "weight": 686592, "pos": 0 }, { "id": -37, "weight": 457851, "pos": 1 }, { "id": -55, "weight": 915699, "pos": 2 }, { "id": -5, "weight": 514924, "pos": 3 }, { "id": -8, "weight": 514924, "pos": 4 }, { "id": -50, "weight": 457851, "pos": 5 }, { "id": -27, "weight": 228930, "pos": 6 } ] }, { "id": -4, "name": "gallo", "type_id": 1, "type_name": "host", "weight": 686592, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 228864, "pos": 0 }, { "id": 1, "weight": 228864, "pos": 1 }, { "id": 8, "weight": 228864, "pos": 2 } ] }, { "id": -5, "name": "joy", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 57212, "pos": 0 }, { "id": 12, "weight": 57212, "pos": 1 }, { "id": 13, "weight": 57212, "pos": 2 }, { "id": 16, "weight": 57212, "pos": 3 }, { "id": 17, "weight": 228864, "pos": 4 }, { "id": 43, "weight": 57212, "pos": 5 } ] }, { "id": -6, "name": "john", "type_id": 1, "type_name": "host", "weight": 457728, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 228864, "pos": 0 }, { "id": 3, "weight": 228864, "pos": 1 } ] }, { "id": -7, "name": "joy~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 5, "weight": 57212, "pos": 0 }, { "id": 12, "weight": 57212, "pos": 1 }, { "id": 13, "weight": 57212, "pos": 2 }, { "id": 16, "weight": 57212, "pos": 3 }, { "id": 17, "weight": 228864, "pos": 4 }, { "id": 43, "weight": 57212, "pos": 5 } ] }, { "id": -8, "name": "jeep", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 18, "weight": 57212, "pos": 0 }, { "id": 19, "weight": 57212, "pos": 1 }, { "id": 20, "weight": 57212, "pos": 2 }, { "id": 21, "weight": 57212, "pos": 3 }, { "id": 22, "weight": 228864, "pos": 4 }, { "id": 45, "weight": 57212, "pos": 5 } ] }, { "id": -9, "name": "john~ssd", "type_id": 1, "type_name": "host", "weight": 457728, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 2, "weight": 228864, "pos": 0 }, { "id": 3, "weight": 228864, "pos": 1 } ] }, { "id": -10, "name": "gold", "type_id": 1, "type_name": "host", "weight": 915456, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 7, "weight": 228864, "pos": 0 }, { "id": 9, "weight": 228864, "pos": 1 }, { "id": 10, "weight": 228864, "pos": 2 }, { "id": 15, "weight": 228864, "pos": 3 } ] }, { "id": -11, "name": "marack3", "type_id": 2, "type_name": "rack", "weight": 3604737, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -52, "weight": 514924, "pos": 0 }, { "id": -46, "weight": 514924, "pos": 1 }, { "id": -43, "weight": 514924, "pos": 2 }, { "id": -10, "weight": 915456, "pos": 3 }, { "id": -41, "weight": 457851, "pos": 4 }, { "id": -16, "weight": 457728, "pos": 5 }, { "id": -12, "weight": 228930, "pos": 6 } ] }, { "id": -12, "name": "innes", "type_id": 1, "type_name": "host", "weight": 228930, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 42, "weight": 114465, "pos": 0 }, { "id": 47, "weight": 114465, "pos": 1 } ] }, { "id": -13, "name": "helm", "type_id": 1, "type_name": "host", "weight": 915456, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 28, "weight": 228864, "pos": 0 }, { "id": 30, "weight": 228864, "pos": 1 }, { "id": 31, "weight": 228864, "pos": 2 }, { "id": 11, "weight": 228864, "pos": 3 } ] }, { "id": -14, "name": "gallo~ssd", "type_id": 1, "type_name": "host", "weight": 686592, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 0, "weight": 228864, "pos": 0 }, { "id": 1, "weight": 228864, "pos": 1 }, { "id": 8, "weight": 228864, "pos": 2 } ] }, { "id": -15, "name": "jeep~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 18, "weight": 57212, "pos": 0 }, { "id": 19, "weight": 57212, "pos": 1 }, { "id": 20, "weight": 57212, "pos": 2 }, { "id": 21, "weight": 57212, "pos": 3 }, { "id": 22, "weight": 228864, "pos": 4 }, { "id": 45, "weight": 57212, "pos": 5 } ] }, { "id": -16, "name": "jack", "type_id": 1, "type_name": "host", "weight": 457728, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 4, "weight": 228864, "pos": 0 }, { "id": 6, "weight": 228864, "pos": 1 } ] }, { "id": -17, "name": "jazz", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 23, "weight": 57212, "pos": 0 }, { "id": 24, "weight": 57212, "pos": 1 }, { "id": 25, "weight": 57212, "pos": 2 }, { "id": 26, "weight": 57212, "pos": 3 }, { "id": 36, "weight": 228864, "pos": 4 }, { "id": 44, "weight": 57212, "pos": 5 } ] }, { "id": -18, "name": "jazz~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 23, "weight": 57212, "pos": 0 }, { "id": 24, "weight": 57212, "pos": 1 }, { "id": 25, "weight": 57212, "pos": 2 }, { "id": 26, "weight": 57212, "pos": 3 }, { "id": 36, "weight": 228864, "pos": 4 }, { "id": 44, "weight": 57212, "pos": 5 } ] }, { "id": -19, "name": "juju", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 37, "weight": 57212, "pos": 0 }, { "id": 38, "weight": 57212, "pos": 1 }, { "id": 39, "weight": 57212, "pos": 2 }, { "id": 40, "weight": 57212, "pos": 3 }, { "id": 41, "weight": 228864, "pos": 4 }, { "id": 46, "weight": 57212, "pos": 5 } ] }, { "id": -20, "name": "juju~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 37, "weight": 57212, "pos": 0 }, { "id": 38, "weight": 57212, "pos": 1 }, { "id": 39, "weight": 57212, "pos": 2 }, { "id": 40, "weight": 57212, "pos": 3 }, { "id": 41, "weight": 228864, "pos": 4 }, { "id": 46, "weight": 57212, "pos": 5 } ] }, { "id": -21, "name": "innes~ssd", "type_id": 1, "type_name": "host", "weight": 228930, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 42, "weight": 114465, "pos": 0 }, { "id": 47, "weight": 114465, "pos": 1 } ] }, { "id": -22, "name": "lc01", "type_id": 1, "type_name": "host", "weight": 228930, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 50, "weight": 114465, "pos": 0 }, { "id": 51, "weight": 114465, "pos": 1 } ] }, { "id": -24, "name": "jack~ssd", "type_id": 1, "type_name": "host", "weight": 457728, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 4, "weight": 228864, "pos": 0 }, { "id": 6, "weight": 228864, "pos": 1 } ] }, { "id": -26, "name": "lc01~ssd", "type_id": 1, "type_name": "host", "weight": 228930, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 50, "weight": 114465, "pos": 0 }, { "id": 51, "weight": 114465, "pos": 1 } ] }, { "id": -27, "name": "lc02", "type_id": 1, "type_name": "host", "weight": 228930, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 52, "weight": 114465, "pos": 0 }, { "id": 73, "weight": 114465, "pos": 1 } ] }, { "id": -28, "name": "gold~ssd", "type_id": 1, "type_name": "host", "weight": 915456, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 7, "weight": 228864, "pos": 0 }, { "id": 9, "weight": 228864, "pos": 1 }, { "id": 10, "weight": 228864, "pos": 2 }, { "id": 15, "weight": 228864, "pos": 3 } ] }, { "id": -29, "name": "lc02~ssd", "type_id": 1, "type_name": "host", "weight": 228930, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 52, "weight": 114465, "pos": 0 }, { "id": 73, "weight": 114465, "pos": 1 } ] }, { "id": -30, "name": "marack~ssd", "type_id": 2, "type_name": "rack", "weight": 3776771, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -14, "weight": 686592, "pos": 0 }, { "id": -40, "weight": 457851, "pos": 1 }, { "id": -56, "weight": 915699, "pos": 2 }, { "id": -7, "weight": 514924, "pos": 3 }, { "id": -15, "weight": 514924, "pos": 4 }, { "id": -53, "weight": 457851, "pos": 5 }, { "id": -29, "weight": 228930, "pos": 6 } ] }, { "id": -34, "name": "marack2~ssd", "type_id": 2, "type_name": "rack", "weight": 3604737, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -35, "weight": 915456, "pos": 0 }, { "id": -20, "weight": 514924, "pos": 1 }, { "id": -51, "weight": 514924, "pos": 2 }, { "id": -18, "weight": 514924, "pos": 3 }, { "id": -47, "weight": 457851, "pos": 4 }, { "id": -9, "weight": 457728, "pos": 5 }, { "id": -26, "weight": 228930, "pos": 6 } ] }, { "id": -35, "name": "helm~ssd", "type_id": 1, "type_name": "host", "weight": 915456, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 28, "weight": 228864, "pos": 0 }, { "id": 30, "weight": 228864, "pos": 1 }, { "id": 31, "weight": 228864, "pos": 2 }, { "id": 11, "weight": 228864, "pos": 3 } ] }, { "id": -37, "name": "kc01", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 82, "weight": 57232, "pos": 0 }, { "id": 83, "weight": 57232, "pos": 1 }, { "id": 84, "weight": 57232, "pos": 2 }, { "id": 85, "weight": 228923, "pos": 3 }, { "id": 86, "weight": 57232, "pos": 4 } ] }, { "id": -38, "name": "marack3~ssd", "type_id": 2, "type_name": "rack", "weight": 3604737, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -54, "weight": 514924, "pos": 0 }, { "id": -48, "weight": 514924, "pos": 1 }, { "id": -45, "weight": 514924, "pos": 2 }, { "id": -28, "weight": 915456, "pos": 3 }, { "id": -42, "weight": 457851, "pos": 4 }, { "id": -24, "weight": 457728, "pos": 5 }, { "id": -21, "weight": 228930, "pos": 6 } ] }, { "id": -39, "name": "default~ssd", "type_id": 6, "type_name": "root", "weight": 10986245, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": -30, "weight": 3776771, "pos": 0 }, { "id": -34, "weight": 3604737, "pos": 1 }, { "id": -38, "weight": 3604737, "pos": 2 } ] }, { "id": -40, "name": "kc01~ssd", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 82, "weight": 57232, "pos": 0 }, { "id": 83, "weight": 57232, "pos": 1 }, { "id": 84, "weight": 57232, "pos": 2 }, { "id": 85, "weight": 228923, "pos": 3 }, { "id": 86, "weight": 57232, "pos": 4 } ] }, { "id": -41, "name": "kc02", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 87, "weight": 57232, "pos": 0 }, { "id": 88, "weight": 57232, "pos": 1 }, { "id": 89, "weight": 57232, "pos": 2 }, { "id": 90, "weight": 228923, "pos": 3 }, { "id": 91, "weight": 57232, "pos": 4 } ] }, { "id": -42, "name": "kc02~ssd", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 87, "weight": 57232, "pos": 0 }, { "id": 88, "weight": 57232, "pos": 1 }, { "id": 89, "weight": 57232, "pos": 2 }, { "id": 90, "weight": 228923, "pos": 3 }, { "id": 91, "weight": 57232, "pos": 4 } ] }, { "id": -43, "name": "jam", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 53, "weight": 57212, "pos": 0 }, { "id": 54, "weight": 57212, "pos": 1 }, { "id": 55, "weight": 57212, "pos": 2 }, { "id": 56, "weight": 57212, "pos": 3 }, { "id": 57, "weight": 57212, "pos": 4 }, { "id": 32, "weight": 228864, "pos": 5 } ] }, { "id": -44, "name": "ks01", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 92, "weight": 57232, "pos": 0 }, { "id": 93, "weight": 57232, "pos": 1 }, { "id": 94, "weight": 57232, "pos": 2 }, { "id": 95, "weight": 57232, "pos": 3 }, { "id": 96, "weight": 228923, "pos": 4 } ] }, { "id": -45, "name": "jam~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 53, "weight": 57212, "pos": 0 }, { "id": 54, "weight": 57212, "pos": 1 }, { "id": 55, "weight": 57212, "pos": 2 }, { "id": 56, "weight": 57212, "pos": 3 }, { "id": 57, "weight": 57212, "pos": 4 }, { "id": 32, "weight": 228864, "pos": 5 } ] }, { "id": -46, "name": "jet", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 58, "weight": 57212, "pos": 0 }, { "id": 59, "weight": 57212, "pos": 1 }, { "id": 60, "weight": 57212, "pos": 2 }, { "id": 61, "weight": 57212, "pos": 3 }, { "id": 62, "weight": 57212, "pos": 4 }, { "id": 33, "weight": 228864, "pos": 5 } ] }, { "id": -47, "name": "ks01~ssd", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 92, "weight": 57232, "pos": 0 }, { "id": 93, "weight": 57232, "pos": 1 }, { "id": 94, "weight": 57232, "pos": 2 }, { "id": 95, "weight": 57232, "pos": 3 }, { "id": 96, "weight": 228923, "pos": 4 } ] }, { "id": -48, "name": "jet~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 58, "weight": 57212, "pos": 0 }, { "id": 59, "weight": 57212, "pos": 1 }, { "id": 60, "weight": 57212, "pos": 2 }, { "id": 61, "weight": 57212, "pos": 3 }, { "id": 62, "weight": 57212, "pos": 4 }, { "id": 33, "weight": 228864, "pos": 5 } ] }, { "id": -49, "name": "jolt", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 63, "weight": 57212, "pos": 0 }, { "id": 64, "weight": 57212, "pos": 1 }, { "id": 65, "weight": 57212, "pos": 2 }, { "id": 66, "weight": 57212, "pos": 3 }, { "id": 67, "weight": 57212, "pos": 4 }, { "id": 34, "weight": 228864, "pos": 5 } ] }, { "id": -50, "name": "ks02", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 97, "weight": 57232, "pos": 0 }, { "id": 98, "weight": 57232, "pos": 1 }, { "id": 99, "weight": 57232, "pos": 2 }, { "id": 100, "weight": 57232, "pos": 3 }, { "id": 101, "weight": 228923, "pos": 4 } ] }, { "id": -51, "name": "jolt~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 63, "weight": 57212, "pos": 0 }, { "id": 64, "weight": 57212, "pos": 1 }, { "id": 65, "weight": 57212, "pos": 2 }, { "id": 66, "weight": 57212, "pos": 3 }, { "id": 67, "weight": 57212, "pos": 4 }, { "id": 34, "weight": 228864, "pos": 5 } ] }, { "id": -52, "name": "jug", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 68, "weight": 57212, "pos": 0 }, { "id": 69, "weight": 57212, "pos": 1 }, { "id": 70, "weight": 57212, "pos": 2 }, { "id": 71, "weight": 57212, "pos": 3 }, { "id": 72, "weight": 57212, "pos": 4 }, { "id": 35, "weight": 228864, "pos": 5 } ] }, { "id": -53, "name": "ks02~ssd", "type_id": 1, "type_name": "host", "weight": 457851, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 97, "weight": 57232, "pos": 0 }, { "id": 98, "weight": 57232, "pos": 1 }, { "id": 99, "weight": 57232, "pos": 2 }, { "id": 100, "weight": 57232, "pos": 3 }, { "id": 101, "weight": 228923, "pos": 4 } ] }, { "id": -54, "name": "jug~ssd", "type_id": 1, "type_name": "host", "weight": 514924, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 68, "weight": 57212, "pos": 0 }, { "id": 69, "weight": 57212, "pos": 1 }, { "id": 70, "weight": 57212, "pos": 2 }, { "id": 71, "weight": 57212, "pos": 3 }, { "id": 72, "weight": 57212, "pos": 4 }, { "id": 35, "weight": 228864, "pos": 5 } ] }, { "id": -55, "name": "lx01", "type_id": 1, "type_name": "host", "weight": 915699, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 14, "weight": 228923, "pos": 0 }, { "id": 27, "weight": 228923, "pos": 1 }, { "id": 29, "weight": 228923, "pos": 2 }, { "id": 48, "weight": 114465, "pos": 3 }, { "id": 49, "weight": 114465, "pos": 4 } ] }, { "id": -56, "name": "lx01~ssd", "type_id": 1, "type_name": "host", "weight": 915699, "alg": "straw2", "hash": "rjenkins1", "items": [ { "id": 14, "weight": 228923, "pos": 0 }, { "id": 27, "weight": 228923, "pos": 1 }, { "id": 29, "weight": 228923, "pos": 2 }, { "id": 48, "weight": 114465, "pos": 3 }, { "id": 49, "weight": 114465, "pos": 4 } ] } ], "rules": [ { "rule_id": 0, "rule_name": "data", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "metadata", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "rbd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ], "tunables": { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }, "choose_args": {} } -- Joe Ryner

1 0

Accessing OSD objects
by Geoffrey Rhodes 24 Feb '23

24 Feb '23

Hello all, I'd really appreciate some input from the more knowledgeable here. Is there a way I can access OSD objects if I have a BlueFS replay error? This error prevents me starting the OSD and also throws an error if I try using the bluestore or objectstore tools. - I can however run a ceph-bluestore-tool show-label without issue. I'm hoping there is another way or possibly a way to purge this log that I can still access the objects on this OSD. If deleting this reply log will help (even with some data loss) I'm happy to try it. This has caused a PG to go inactive and I'm considering deleting the PG and force re-creating it. - Saw this mentioned as a last resort option. Below is a snip of where things go wrong. - I don't know if there is even a chance or is this an unrecoverable state? 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/031664.sst to 29549 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/031665.sst to 29550 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/031666.sst to 29551 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/CURRENT to 29543 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/IDENTITY to 5 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/LOCK to 2 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/MANIFEST-031657 to 29542 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/OPTIONS-031645 to 29529 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_link db/OPTIONS-031660 to 29545 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_dir_create db.slow 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _replay 0x0: op_jump seq 5204712 offset 0x20000 2023-01-25T10:05:26.543+0000 7fa773a14240 10 bluefs _read h 0x55d2f1cfdb80 0x10000~10000 from file(ino 1 size 0x0 mtime 2022-10-07T17:55:34.189440+0000 allocated 420000 alloc_commit 420000 extents [1:0x1770170000~20000,1:0x53d1e900000~400000]) 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _read left 0x10000 len 0x10000 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _read got 65536 2023-01-25T10:05:26.543+0000 7fa773a14240 10 bluefs _read h 0x55d2f1cfdb80 0x20000~1000 from file(ino 1 size 0x20000 mtime 2022-10-07T17:55:34.189440+0000 allocated 420000 alloc_commit 420000 extents [1:0x1770170000~20000,1:0x53d1e900000~400000]) 2023-01-25T10:05:26.543+0000 7fa773a14240 20 bluefs _read fetching 0x0~100000 of 1:0x53d1e900000~400000 2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _read left 0x100000 len 0x1000 2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _read got 4096 2023-01-25T10:05:26.547+0000 7fa773a14240 10 bluefs _replay 0x20000: txn(seq 5204713 len 0x55 crc 0x81f48b1c) 2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _replay 0x20000: op_file_update file(ino 29551 size 0x0 mtime 2022-10-07T17:55:34.151007+0000 allocated 0 alloc_commit 0 extents []) 2023-01-25T10:05:26.547+0000 7fa773a14240 20 bluefs _replay 0x20000: op_dir_link db/031666.sst to 29551 2023-01-25T10:05:26.555+0000 7fa773a14240 -1 /build/ceph-17.2.5/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_replay(bool, bool)' thread 7fa773a14240 time 2023-01-25T10:05:26.551808+0000 /build/ceph-17.2.5/src/os/bluestore/BlueFS.cc: 1419: FAILED ceph_assert(r == q->second->file_map.end()) Kind regards Geoff

1 0

Daily failed capability releases, slow ops, fully stuck IO
by Kuhring, Mathias 24 Feb '23

24 Feb '23

Dear Ceph community, since about two or three weeks, we have CephFS clients regularly failing to respond to capability releases accompanied OSD slow ops. By now, this happens daily every time clients get more active (e.g. during nightly backups). We mostly observe it with a handful of highly active clients, so correlating with IO volume. But we have over 250 clients which mount the CephFS and plan to get them all more active soon. What's worrying me further, it doesn't seem to effect only the clients which fail to respond to the capability release. But also other clients get just stuck accessing data on the CephFS. So far I've been tracking down the corresponding OSDs via the client (`cat /sys/kernel/debug/ceph/*/osdc`) and restarted them one by one. But since this is now a regular/systemic issue, this is obviously no sustainable solution. This would be usually a handful of OSDs per client and I couldn't observe any particular pattern of involved OSDs, yet. Our cluster still runs on CentOS 7 with kernel 3.10.0-1160.42.2.el7.x86_64 using cephadm with ceph version 17.2.1 (ec95624474b1871a821a912b8c3af68f8f8e7aa1) quincy (stable). Most active clients are currently on kernel versions such as: 4.18.0-348.el8.0.2.x86_64, 4.18.0-348.2.1.el8_5.x86_64, 4.18.0-348.7.1.el8_5.x86_64 I picked up some logging ideas from an older issue with similar symptoms: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CKTIM6LF274… But this has been already fixed in the kernel client and I don't have similar things in the logs. But also I'm not sure if the things I digging up in the logs are actually useful. And if I'm actually looking in the right places. So, I enabled "debug_ms 1" for the OSDs as suggested in the other thread. But this filled up our host disks pretty fast, leading to e.g. monitors crashing. I disabled the debug messages again and trimmed logs to free up space. But I made copies of two OSD logs files which were involved in a capability release / slow requests issue. They are quite big now (~3GB) and even if I remove things like ping stuff, I have more than 1 million lines just for the morning until the disk space was full (around 7 hours). So now I'm wondering how to filter/look for the right things here. When I grep for "error", I get a few of these messages: {"log":"debug 2023-02-22T06:18:08.113+0000 7f15c5fff700 1 -- [v2:192.168.1.13:6881/4149819408,v1:192.168.1.13:6884/4149819408] \u003c== osd.161 v2:192.168.1.31:6835/1012436344 182573 ==== pg_update_log_missing(3.1a6s2 epoch 646235/644895 rep_tid 1014320 entries 646235'7672108 (0'0) error 3:65836dde:::10016e9b7c8.00000000:head by mds.0.1221974:8515830 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 646178'7662340 roll_forward_to 646192'7672106) v3 ==== 261+0+0 (crc 0 0 0) 0x562d55e52380 con 0x562d8a2de400\n","stream":"stderr","time":"2023-02-22T06:18:08.115002765Z"} And if I grep for "failed", I get a couple of those: {"log":"debug 2023-02-22T06:15:25.242+0000 7f58bbf7c700 1 -- [v2:172.16.62.11:6829/3509070161,v1:172.16.62.11:6832/3509070161] \u003e\u003e 172.16.62.10:0/3127362489 conn(0x55ba06bf3c00 msgr2=0x55b9ce07e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed\n","stream":"stderr","time":"2023-02-22T06:15:25.243808392Z"} {"log":"debug 2023-02-22T06:15:25.242+0000 7f58bbf7c700 1 --2- [v2:172.16.62.11:6829/3509070161,v1:172.16.62.11:6832/3509070161] \u003e\u003e 172.16.62.10:0/3127362489 conn(0x55ba06bf3c00 0x55b9ce07e580 crc :-1 s=READY pgs=2096664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted)\n","stream":"stderr","time":"2023-02-22T06:15:25.243813528Z"} Not sure, if they are related to the issue. In the kernel logs of the client (dmesg, journalctl or /var/log/messages), there seem to be no errors or any stack traces in the relevant time periods. The only thing I can see is me restarting the relevant OSDs: [Mi Feb 22 07:29:59 2023] libceph: osd90 down [Mi Feb 22 07:30:34 2023] libceph: osd90 up [Mi Feb 22 07:31:55 2023] libceph: osd93 down [Mi Feb 22 08:37:50 2023] libceph: osd93 up I noticed a socket closed for another client, but I assume that's more related to monitors failing due to full disks: [Mi Feb 22 05:59:52 2023] libceph: mon2 (1)172.16.62.12:6789 socket closed (con state OPEN) [Mi Feb 22 05:59:52 2023] libceph: mon2 (1)172.16.62.12:6789 session lost, hunting for new mon [Mi Feb 22 05:59:52 2023] libceph: mon3 (1)172.16.62.13:6789 session established I would appreciate if anybody has a suggestion where I should look next. Thank you for your help Best Wishes, Mathias

1 0

MDS stuck in "up:replay"
by Thomas Widhalm 23 Feb '23

23 Feb '23

Hi, I'm really lost with my Ceph system. I built a small cluster for home usage which has two uses for me: I want to replace an old NAS and I want to learn about Ceph so that I have hands-on experience. We're using it in our company but I need some real-life experience without risking any company or customers data. That's my preferred way of learning. The cluster consists of 3 Raspberry Pis plus a few VMs running on Proxmox. I'm not using Proxmox' built in Ceph because I want to focus on Ceph and not just use it as a preconfigured tool. All hosts are running Fedora (x86_64 and arm64) and during an Upgrade from F36 to F37 my cluster suddenly showed all PGs as unavailable. I worked nearly a week to get it back online and I learned a lot about Ceph management and recovery. The cluster is back but I still can't access my data. Maybe you can help me? Here are my versions: [ceph: root@ceph04 /]# ceph versions { "mon": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 5 }, "mds": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 4 }, "overall": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 15 } } Here's MDS status output of one MDS: [ceph: root@ceph04 /]# ceph tell mds.mds01.ceph05.pqxmvt status 2023-01-14T15:30:28.607+0000 7fb9e17fa700 0 client.60986454 ms_handle_reset on v2:192.168.23.65:6800/2680651694 2023-01-14T15:30:28.640+0000 7fb9e17fa700 0 client.60986460 ms_handle_reset on v2:192.168.23.65:6800/2680651694 { "cluster_fsid": "ff6e50de-ed72-11ec-881c-dca6325c2cc4", "whoami": 0, "id": 60984167, "want_state": "up:replay", "state": "up:replay", "fs_name": "cephfs", "replay_status": { "journal_read_pos": 0, "journal_write_pos": 0, "journal_expire_pos": 0, "num_events": 0, "num_segments": 0 }, "rank_uptime": 1127.54018615, "mdsmap_epoch": 98056, "osdmap_epoch": 12362, "osdmap_epoch_barrier": 0, "uptime": 1127.957307273 } It's staying like that for days now. If there was a counter moving, I just would wait but it doesn't change anything and alle stats says, the MDS aren't working at all. The symptom I have is that Dashboard and all other tools I use say, it's more or less ok. (Some old messages about failed daemons and scrubbing aside). But I can't mount anything. When I try to start a VM that's on RDS I just get a timeout. And when I try to mount a CephFS, mount just hangs forever. Whatever command I give MDS or journal, it just hangs. The only thing I could do, was take all CephFS offline, kill the MDS's and do a "ceph fs reset <fs name> --yes-i-really-mean-it". After that I rebooted all nodes, just to be sure but I still have no access to data. Could you please help me? I'm kinda desperate. If you need any more information, just let me know. Cheers, Thomas -- Thomas Widhalm Lead Systems Engineer NETWAYS Professional Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB34510 https://www.netways.de | thomas.widhalm(a)netways.de ** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** New at NWS: Managed Database - https://nws.netways.de/managed-database ** ** NETWAYS Web Services - https://nws.netways.de **

7 23

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

slow replication of large buckets
by Glaza 23 Feb '23

23 Feb '23

Hi Cephers, We have two octopus 15.2.17 clusters in a multisite configuration. Every once in a while we have to perform a bucket reshard (most recently on 613 shards) and this practically kills our replication for a few days. Does anyone know of any priority mechanics within sync to give priority to other buckets and/or lower them? Are there any improvements to this in higher versions of ceph that we could take advantage of if we upgrade the cluster (I haven't found any)? How to safely perform the increase of rgw_data_log_num_shards, because the documentation only says: "The values of rgw_data_log_num_shards and rgw_md_log_max_shards should not be changed after sync has started." Does this mean that I should block access to the cluster, wait until sync is caught up with source/master, change this value, restart rgw and unblock access? Kind Regards, Tom

1 0

increasing PGs OOM kill SSD OSDs (octopus) - unstable OSD behavior
by Boris Behrens 23 Feb '23

23 Feb '23

Hi, today I wanted to increase the PGs from 2k -> 4k and random OSDs went offline in the cluster. After some investigation we saw, that the OSDs got OOM killed (I've seen a host that went from 90GB used memory to 190GB before OOM kills happen). We have around 24 SSD OSDs per host and 128GB/190GB/265GB memory in these hosts. All of them experienced OOM kills. All hosts are octopus / ubuntu 20.04. And on every step new OSDs crashed with OOM. (We now set the pg_num/pgp_num to 2516 to stop the process). The OSD logs do not show anything why this might happen. Some OSDs also segfault. I now started to stop all OSDs on a host, and do a "ceph-bluestore-tool repair" and a "ceph-kvstore-tool bluestore-kv compact" on all OSDs. This takes for the 8GB OSDs around 30 minutes. When I start the OSDs I instantly get a lot of slow OPS from all the other OSDs when the OSD come up (the 8TB OSDs take around 10 minutes with "load_pgs". I am unsure what I can do to restore normal cluster performance. Any ideas or suggestions or maybe even known bugs? Maybe a line for what I can search in the logs. Cheers Boris

2 3

setup problem for ingress + SSL for RGW
by Patrick Vranckx 23 Feb '23

23 Feb '23

Hi, Our cluster runs Pacific on Rocky8. We have 3 rgw running on port 7480. I tried to setup an ingress service with a yaml definition of service: no luck service_type: ingress service_id: rgw.myceph.be placement: hosts: - ceph001 - ceph002 - ceph003 spec: backend_service: rgw.myceph.be virtual_ip: 192.168.0.10 frontend_port: 443 monitor_port: 9000 ssl_cert: | -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY----- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- I tried to setup the ingress service with the dashboard... still no luck. I started debugging the problem. 1. Even if I entered the certificate and the private key in the form, CEPH complained about no haproxy.pem.key file. I added manually the file in the container folder definition. Haproxy containers started ! 2. Looking at the monitoring page of HAProxy, I realized that there was no backend server defined. In the form, I selected manually the servers running the rgw. In the container definition folder, the backend definition of haproxy.cfg looks like: ... backend backend option forwardfor balance static-rr option httpchk HEAD / HTTP/1.0 No mention of servers or port 7480 Once again, I added the definition manually : server ceph001 192.168.0.1:7480 check server ceph004 192.168.0.2:7480 check server ceph008 192.168.0.2:7480 check and redeployed the containers. It's working. Any idea ? Patrick

1 0

[Quincy] Module 'devicehealth' has failed: disk I/O error
by Satish Patel 22 Feb '23

22 Feb '23

Folks, Any idea what is going on, I am running 3 node quincy version of openstack and today suddenly i noticed the following error. I found reference link but not sure if that is my issue or not https://tracker.ceph.com/issues/51974 root@ceph1:~# ceph -s cluster: id: cd748128-a3ea-11ed-9e46-c309158fad32 health: HEALTH_ERR 1 mgr modules have recently crashed services: mon: 3 daemons, quorum ceph1,ceph2,ceph3 (age 2d) mgr: ceph1.ckfkeb(active, since 6h), standbys: ceph2.aaptny osd: 9 osds: 9 up (since 2d), 9 in (since 2d) data: pools: 4 pools, 128 pgs objects: 1.18k objects, 4.7 GiB usage: 17 GiB used, 16 TiB / 16 TiB avail pgs: 128 active+clean root@ceph1:~# ceph health HEALTH_ERR Module 'devicehealth' has failed: disk I/O error; 1 mgr modules have recently crashed root@ceph1:~# ceph crash ls ID ENTITY NEW 2023-02-07T00:07:12.739187Z_fcb9cbc9-bb55-4e7c-bf00-945b96469035 mgr.ceph1.ckfkeb * root@ceph1:~# ceph crash info 2023-02-07T00:07:12.739187Z_fcb9cbc9-bb55-4e7c-bf00-945b96469035 { "backtrace": [ " File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 373, in serve\n self.scrape_all()", " File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 425, in scrape_all\n self.put_device_metrics(device, data)", " File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 500, in put_device_metrics\n self._create_device(devid)", " File \"/usr/share/ceph/mgr/devicehealth/module.py\", line 487, in _create_device\n cursor = self.db.execute(SQL, (devid,))", "sqlite3.OperationalError: disk I/O error" ], "ceph_version": "17.2.5", "crash_id": "2023-02-07T00:07:12.739187Z_fcb9cbc9-bb55-4e7c-bf00-945b96469035", "entity_name": "mgr.ceph1.ckfkeb", "mgr_module": "devicehealth", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "OperationalError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "7e506cc2729d5a18403f0373447bb825b42aafa2405fb0e5cfffc2896b093ed8", "timestamp": "2023-02-07T00:07:12.739187Z", "utsname_hostname": "ceph1", "utsname_machine": "x86_64", "utsname_release": "5.15.0-58-generic", "utsname_sysname": "Linux", "utsname_version": "#64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023"

2 1

Problem with IO after renaming File System .data pool
by murilo＠evocorp.com.br 22 Feb '23

22 Feb '23

Good morning everyone. On this Thursday night we went through an accident, where they accidentally renamed the .data pool of a File System making it instantly inaccessible, when renaming it again to the correct name it was possible to mount and list the files, but could not read or write. When trying to write, the FS returned as Read Only, when trying to read it returned Operation not allowed. After a period of breaking my head I tried to mount with the ADMIN user and everything worked correctly. I tried to remove the authentication of the current user through `ceph auth rm`, I created a new user through `ceph fs authorize <fs_name> client.<user> / rw` and it continued the same way, I also tried to recreate it through `ceph auth get-or-create` and nothing different happened, it stayed exactly the same. After setting `allow *` in mon, mds and osd I was able to mount, read and write again with the new user. I can understand why the File System stopped after renaming the pool, what I don't understand is why users are unable to perform operations on FS even with RW or any other user created. What could have happened behind the scenes to not be able to perform IO even with the correct permissions? Or did I apply incorrect permissions that caused this problem? Right now everything is working, I would really like to understand what happened, because I didn't find anything documented about this type of incident.

3 2

17.2.5 ceph fs status: AssertionError
by Robert Sander 22 Feb '23

22 Feb '23

Hi, I have a healthy (test) cluster running 17.2.5: root@cephtest20:~# ceph status cluster: id: ba37db20-2b13-11eb-b8a9-871ba11409f6 health: HEALTH_OK services: mon: 3 daemons, quorum cephtest31,cephtest41,cephtest21 (age 2d) mgr: cephtest22.lqzdnk(active, since 4d), standbys: cephtest32.ybltym, cephtest42.hnnfaf mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 48 osds: 48 up (since 4d), 48 in (since 4M) rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 6 portals active (3 hosts) data: volumes: 1/1 healthy pools: 17 pools, 513 pgs objects: 28.25k objects, 4.7 GiB usage: 26 GiB used, 4.7 TiB / 4.7 TiB avail pgs: 513 active+clean io: client: 4.3 KiB/s rd, 170 B/s wr, 5 op/s rd, 0 op/s wr CephFS is mounted and can be used without any issue. But I get an error when I when querying its status: root@cephtest20:~# ceph fs status Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1757, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/status/module.py", line 159, in handle_fs_status assert metadata AssertionError The dashboard's filesystem page shows no error and displays all information about cephfs. Where does this AssertionError come from? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 3

Ceph Leadership Team Meeting, Feb 22 2023 Minutes
by Ernesto Puerta 22 Feb '23

22 Feb '23

Hi Cephers, These are the minutes of today's meeting (quicker than usual since some CLT members were at Ceph Days NYC): - *[Yuri] Upcoming Releases:* - Pending PRs for Quincy - Sepia Lab still absorbing the PR queue after the past issues - [Ernesto] Github started sending dependabot alerts to devels (previously it was only sent to org admins) - https://github.blog/2023-01-17-dependabot-alerts-are-now-visible-to-more-de… - Most don't necessarily involve a risk (e.g.: Javascript dependency only exploitable in a back-end/node.js server)... - ... but it might still cause some unnecessary concern among devs/users regarding Ceph security status - Current list of vulnerable dependencies: https://github.com/ceph/ceph/security/dependabot - 40% are Dashboard Javascript ones (most could be dismissed since only impact when used on node.js apps) - Remaining ones are: - Python: requirements.txt (not relevant since Python package versions change with every distro and we assume distro-maintainers will fix those) - It might become more relevant when we start packaging Python deps ( https://github.com/ceph/ceph/pull/47501/) - Golang: "/examples/rgw" path (Casey opened https://tracker.ceph.com/issues/58828, but maybe we should just dismiss the alert?) - [Ernesto] Enabling Github Auto-merge feature in the Ceph repo - https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/i… - Use case: - There's a PR with approvals but flaky CI tests (API, make check, ...) (example: https://github.com/ceph/ceph/pull/50201) - We could retrigger tests and come back to the PR page multiple times until all tests pass... - ... Or we just click the "Auto-merge" button, fill out the merge message as usual, and let Github merge it when the CI tests pass. - It'd reduce cognitive load, especially with small PRs (docs, backport PRs) where the overhead of the PR process is more noticeable. - There's still one issue: - Keeping Redmine in sync with Github - It could be done: when clicking the Auto-merge or still requiring reviewers to poll the PR until passed and then updating Redmine (not ideal) - A Github action that update a tracker when Github merges the PR would be very useful - Yuri/Ilya: discussion around backport requirement reverse order (needs-qa label vs. approvals vs. CI tests passing). - Greg pointed out the risks of auto-merge merging PRs with patches submitted after passing requirements or approvals. Auto-merge status should be reset on new commits. - Decision: not to enable it. - Yuri suggested auto-labeling PRs with passing CI, so they better know when to start QA testing. - Separate discussion on CI flakiness & stability and lack of clear points of contact (Kefu and David did that). For unit tests it's clear that affected teams should do that, but for infrastructure issues there's still a vacuum. Kind Regards, Ernesto

1 0

Strange behavior when using storage classes
by Michal Strnad 22 Feb '23

22 Feb '23

Hi all, we encountered some strange behavior when using storage classes for S3 protocol. Some objects end up in a different pool than we would expect. Below is a list of commands used for create an account with replicated storage class, upload some files to the bucket and checked that they were uploaded to the correct location. Most of the files were uploaded to the correct pool, but some were uploaded to the erasure-code pool. Specifically, certain multipart objects (but they have zero size) and stub. The multipart objects probably wouldn't mind so much, but stub is important because we tried to delete it and without it the operations will not go through. Are there any errors in our setting (see zonegroup and zone setttings below)? Can someone please send us a setup that is functional so we can build it accordingly? We have a Pacific version. === Create the account === radosgw-admin user create --tenant vo_du_test --uid s3_test_tomash_ec_replicated --display-name s3_test_tomash_ec_replicated --storage-class "" --placement-id "EC_replicated" --tags "" radosgw-admin user info --tenant vo_du_test --uid s3_test_tomash_ec_replicated | jq '.default_placement,.default_storage_class,.placement_tags' "EC_replicated" "" [] === Create the bucket with replicated storage class === s3cmd mb s3://b7-user-ec-stcl-replicated --storage-class=replicated Bucket 's3://b7-user-ec-stcl-replicated/' created s3cmd -c ~/.s3cfg/.s3cfg_clx_vo_du_test_s3_test_tomash_ec_replicated put 20MiB.dat s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat --storage-class=replicated upload: '20MiB.dat' -> 's3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat' [part 1 of 2, 15MB] [1 of 1] 15728640 of 15728640 100% in 0s 20.86 MB/s done upload: '20MiB.dat' -> 's3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat' [part 2 of 2, 5MB] [1 of 1] 5242880 of 5242880 100% in 0s 18.34 MB/s done s3cmd -c ~/.s3cfg/.s3cfg_clx_vo_du_test_s3_test_tomash_ec_replicated info s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat (object): File size: 20971520 Storage: replicated ... ACL: s3_test_tomash_ec_replicated: FULL_CONTROL ... === Checking where objects are stored === rados -p storage-clx.rgw.replicated.data ls | grep _b7_repl 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 ceph_mon:# rados -p storage-clx.rgw.EC.data ls | grep _b7_repl 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat ceph_mon:# rados stat -p storage-clx.rgw.EC.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat mtime 2023-01-24T12:46:48.000000+0100, size 0 ceph_mon:# rados stat -p storage-clx.rgw.EC.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 mtime 2023-01-24T12:46:48.000000+0100, size 0 ceph_mon:# rados stat -p storage-clx.rgw.EC.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 mtime 2023-01-24T12:46:48.000000+0100, size 0 ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1 mtime 2023-01-24T12:46:47.000000+0100, size 4194304 rados stat -p storage-clx.rgw.replicated.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1 storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1 mtime 2023-01-24T12:46:48.000000+0100, size 4194304 ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2 storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2 mtime 2023-01-24T12:46:48.000000+0100, size 4194304 ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3 storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3 mtime 2023-01-24T12:46:48.000000+0100, size 3145728 ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2 mtime 2023-01-24T12:46:48.000000+0100, size 4194304 ceph_mon:# rados stat -p storage-clx.rgw.replicated.data 68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1 storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1 mtime 2023-01-24T12:46:48.000000+0100, size 1048576 === Zonegroup setting === radosgw-admin zonegroup get { "id": "kda7dadb-7bbc-4197-b070-147f76e74717", "name": "storage", "api_name": "storage", "is_master": "true", "endpoints": [ "https://s3.clx.domain.cz" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "69922cb9-d006-4b86-b8f0-3722d952fd33", "zones": [ { "id": "24654cb9-d006-4b86-b8f0-3722d952fd33", "name": "storage-clx", "endpoints": [ "https://s3.clx.domain.cz" ], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "EC_replicated", "tags": [], "storage_classes": [ "EC", "STANDARD", "replicated" ] }, { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] }, { "name": "replicated", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "87a9217-7e8d-4a1b-b508-eb634cc55647", "sync_policy": { "groups": [] } } === Zone setting === radosgw-admin zone get { "id": "24654cb9-d006-4b86-b8f0-3722d952fd33", "name": "storage-clx", "domain_root": "storage-clx.rgw.meta:root", "control_pool": "storage-clx.rgw.control", "gc_pool": "storage-clx.rgw.log:gc", "lc_pool": "storage-clx.rgw.log:lc", "log_pool": "storage-clx.rgw.log", "intent_log_pool": "storage-clx.rgw.log:intent", "usage_log_pool": "storage-clx.rgw.log:usage", "roles_pool": "storage-clx.rgw.meta:roles", "reshard_pool": "storage-clx.rgw.log:reshard", "user_keys_pool": "storage-clx.rgw.meta:users.keys", "user_email_pool": "storage-clx.rgw.meta:users.email", "user_swift_pool": "storage-clx.rgw.meta:users.swift", "user_uid_pool": "storage-clx.rgw.meta:users.uid", "otp_pool": "storage-clx.rgw.otp", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "EC_replicated", "val": { "index_pool": "storage-clx.rgw.EC.index", "storage_classes": { "STANDARD": { "data_pool": "storage-clx.rgw.EC.data" }, "replicated": { "data_pool": "storage-clx.rgw.replicated.data" } }, "data_extra_pool": "storage-clx.rgw.EC.non-ec", "index_type": 0 } }, { "key": "default-placement", "val": { "index_pool": "storage-clx.rgw.EC.index", "storage_classes": { "STANDARD": { "data_pool": "storage-clx.rgw.EC.data" } }, "data_extra_pool": "storage-clx.rgw.EC.non-ec", "index_type": 0 } }, { "key": "replicated", "val": { "index_pool": "storage-clx.rgw.replicated.index", "storage_classes": { "STANDARD": { "data_pool": "storage-clx.rgw.replicated.data" } }, "data_extra_pool": "", "index_type": 0 } } ], "realm_id": "87a9217-7e8d-4a1b-b508-eb634cc55647", "notif_pool": "storage-clx.rgw.log:notif" } Thank you Regards, Michal Strnad

1 0

Re: Stuck OSD service specification - can't remove
by Eugen Block 21 Feb '23

21 Feb '23

Hi, did you ever resolve that? I'm stuck with the same "deleting" service in 'ceph orch ls' and found your thread. Thanks, Eugen

1 0

Missing keyrings on upgraded cluster
by Eugen Block 20 Feb '23

20 Feb '23

Hi *, I was playing around on an upgraded test cluster (from N to Q), current version: "overall": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 18 } I tried to replace an OSD after destroying it with 'ceph orch osd rm osd.5 --replace'. The OSD was drained successfully and marked as "destroyed" as expected, the zapping also worked. At this point I didn't have an osd spec in place because all OSDs were adopted during the upgrade process. So I created a new spec which was not applied successfully (I'm wondering if there's another/new issue with ceph-volume, but that's not the focus here), so I tried it manually with 'cephadm ceph-volume lvm create'. I'll add the output at the end for a better readability. Apparently, there's no boostrap-osd keyring for cephadm so it can't search the desired osd_id in the osd tree, the command it tries is this: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json In the local filesystem the required keyring is present, though: nautilus:~ # cat /var/lib/ceph/bootstrap-osd/ceph.keyring [client.bootstrap-osd] key = AQBOCbpgixIsOBAAgBzShsFg/l1bOze4eTZHug== caps mgr = "allow r" caps mon = "profile bootstrap-osd" Is there something missing during the adoption process? Or are the docs lacking some upgrade info? I found a section about putting keyrings under management [1], but I'm not sure if that's what's missing here. Any insights are highly appreciated! Thanks, Eugen [1] https://docs.ceph.com/en/quincy/cephadm/operations/#putting-a-keyring-under… ---snip--- nautilus:~ # cephadm ceph-volume lvm create --osd-id 5 --data /dev/sde --block.db /dev/sdb --block.db-size 5G Inferring fsid <FSID> Using recent ceph image <LOCAL_REGISTRY>/ceph/ceph@sha256:af50ec26db7ee177e1ec1b553a0d6a9dbad2c3cc0da2f8f46d012184a79d4f92 Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=<LOCAL_REGISTRY>/ceph/ceph@sha256:af50ec26db7ee177e1ec1b553a0d6a9dbad2c3cc0da2f8f46d012184a79d4f92 -e NODE_NAME=nautilus -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/<FSID>:/var/run/ceph:z -v /var/log/ceph/<FSID>:/var/log/ceph:z -v /var/lib/ceph/<FSID>/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpuydvbhuk:/etc/ceph/ceph.conf:z <LOCAL_REGISTRY>/ceph/ceph@sha256:af50ec26db7ee177e1ec1b553a0d6a9dbad2c3cc0da2f8f46d012184a79d4f92 lvm create --osd-id 5 --data /dev/sde --block.db /dev/sdb --block.db-size 5G /usr/bin/podman: stderr time="2023-02-20T09:02:49+01:00" level=warning msg="Path \"/etc/SUSEConnect\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping" /usr/bin/podman: stderr time="2023-02-20T09:02:49+01:00" level=warning msg="Path \"/etc/zypp/credentials.d/SCCcredentials\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping" /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd tree -f json /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.848+0000 7fd255e30700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.848+0000 7fd255e30700 -1 AuthRegistry(0x7fd250060d50) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.852+0000 7fd255e30700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.852+0000 7fd255e30700 -1 AuthRegistry(0x7fd250060d50) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.856+0000 7fd255e30700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.856+0000 7fd255e30700 -1 AuthRegistry(0x7fd250065910) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.856+0000 7fd255e30700 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory /usr/bin/podman: stderr stderr: 2023-02-20T08:02:50.856+0000 7fd255e30700 -1 AuthRegistry(0x7fd255e2eea0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx /usr/bin/podman: stderr stderr: [errno 2] RADOS object not found (error connecting to the cluster) /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/podman: stderr self.main(self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr return f(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 77, in main /usr/bin/podman: stderr self.create(args) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create /usr/bin/podman: stderr prepare_step.safe_prepare(args) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare /usr/bin/podman: stderr self.prepare() /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 292, in prepare /usr/bin/podman: stderr self.osd_id = prepare_utils.create_id(osd_fsid, json.dumps(secrets), osd_id=self.args.osd_id) /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 166, in create_id /usr/bin/podman: stderr if osd_id_available(osd_id): /usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 204, in osd_id_available /usr/bin/podman: stderr raise RuntimeError('Unable check if OSD id exists: %s' % osd_id) /usr/bin/podman: stderr RuntimeError: Unable check if OSD id exists: 5 Traceback (most recent call last): File "/usr/sbin/cephadm", line 9170, in <module> main() File "/usr/sbin/cephadm", line 9158, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 1917, in _infer_config return func(ctx) File "/usr/sbin/cephadm", line 1877, in _infer_fsid return func(ctx) File "/usr/sbin/cephadm", line 1945, in _infer_image return func(ctx) File "/usr/sbin/cephadm", line 1835, in _validate_fsid return func(ctx) File "/usr/sbin/cephadm", line 5294, in command_ceph_volume out, err, code = call_throws(ctx, c.run_cmd()) File "/usr/sbin/cephadm", line 1637, in call_throws raise RuntimeError('Failed command: %s' % ' '.join(command)) RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=<LOCAL_REGISTRY>/ceph/ceph@sha256:af50ec26db7ee177e1ec1b553a0d6a9dbad2c3cc0da2f8f46d012184a79d4f92 -e NODE_NAME=nautilus -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/<FSID>:/var/run/ceph:z -v /var/log/ceph/<FSID>:/var/log/ceph:z -v /var/lib/ceph/<FSID>/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpuydvbhuk:/etc/ceph/ceph.conf:z <LOCAL_REGISTRY>/ceph/ceph@sha256:af50ec26db7ee177e1ec1b553a0d6a9dbad2c3cc0da2f8f46d012184a79d4f92 lvm create --osd-id 5 --data /dev/sde --block.db /dev/sdb --block.db-size 5G ---snip---

2 7

Very slow snaptrim operations blocking client I/O
by Victor Rodriguez 20 Feb '23

20 Feb '23

Hello, Asking for help with an issue. Maybe someone has a clue about what's going on. Using ceph 15.2.17 on Proxmox 7.3. A big VM had a snapshot and I removed it. A bit later, nearly half of the PGs of the pool entered snaptrim and snaptrim_wait state, as expected. The problem is that such operations ran extremely slow and client I/O was nearly nothing, so all VMs in the cluster got stuck as they could not I/O to the storage. Taking and removing big snapshots is a normal operation that we do often and this is the first time I see this issue in any of my clusters. Disks are all Samsung PM1733 and network is 25G. It gives us plenty of performance for the use case and never had an issue with the hardware. Both disk I/O and network I/O was very low. Still, client I/O seemed to get queued forever. Disabling snaptrim (ceph osd set nosnaptrim) stops any active snaptrim operation and client I/O resumes back to normal. Enabling snaptrim again makes client I/O to almost halt again. I've been playing with some settings: ceph tell 'osd.*' injectargs '--osd-max-trimming-pgs 1' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep 30' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep-ssd 30' ceph tell 'osd.*' injectargs '--osd-pg-max-concurrent-snap-trims 1' None really seemed to help. Also tried restarting OSD services. This cluster was upgraded from 14.2.x to 15.2.17 a couple of months. Is there any setting that must be changed which may cause this problem? I have scheduled a maintenance window, what should I look for to diagnose this problem? Any help is very appreciated. Thanks in advance. Victor

8 18

ceph-iscsi-cli: cannot remove duplicated gateways.
by luckydog xf 20 Feb '23

20 Feb '23

Hi, please see the output below. ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain is the one who is being messed up with a wrong hostname. I want to delete it. /iscsi-target...-igw/gateways> ls o- gateways .................................................................................................. [Up: 2/3, Portals: 3] o- ceph-iscsi-gw-1.ipa.pthl.hk ............................................................................. [172.16.202.251 (UP)] o- ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain .............................................. [172.16.202.251 (UNAUTHORIZED)] o- ceph-iscsi-gw-2.ipa.pthl.hk ............................................................................. [172.16.202.252 (UP)] /iscsi-target...-igw/gateways> delete gateway_name=ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain confirm=true Deleting gateway, ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain Could not contact ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain. If the gateway is permanently down. Use confirm=true to force removal. WARNING: Forcing removal of a gateway that can still be reached by an initiator may result in data corruption. /iscsi-target...-igw/gateways> /iscsi-target...-igw/gateways> delete gateway_name=ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain confirm=true Deleting gateway, ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain Failed : Unhandled exception: list.remove(x): x not in list However ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain is still there. Version info is ceph-iscsi-3.5-1.el8cp.noarch on RHEL 8.4. /iscsi-target...-igw/gateways> ls o- gateways .................................................................................................. [Up: 2/3, Portals: 3] o- ceph-iscsi-gw-1.ipa.pthl.hk ............................................................................. [172.16.202.251 (UP)] o- ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain ................................................... [172.16.202.251 (UNKNOWN)] o- ceph-iscsi-gw-2.ipa.pthl.hk ............................................................................. [172.16.202.252 (UP)] /iscsi-target...-igw/gateways> delete ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain confirm=true Deleting gateway, ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain Failed : Unhandled exception: list.remove(x): x not in list However ceph-iscsi-gw-1.ipa.pthl.hklocalhost.localdomain is still there. Please help, thanks.

2 8

Removing failing OSD with cephadm?
by Matt Larson 18 Feb '23

18 Feb '23

I have an OSD that is causing slow ops, and appears to be backed by a failing drive according to smartctl outputs. I am using cephadm, and wondering what is the best way to remove this drive from the cluster and proper steps to replace the disk? Mark the osd.35 as out. `sudo ceph osd out osd.35` Then mark osd.35 as down. `sudo ceph osd down osd.35` The OSD is marked as out, but it does come back up after a couple of seconds. I do not know if that is a problem or to just let the drive stay online as long as it lasts during the removal from the cluster. After the recovery completes, I would then `destroy` the osd: `ceph osd destroy {id} --yes-i-really-mean-it` (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/) Besides checking steps above, my question now is ..* If the drive is acting very slow and causing slow ops, should I be trying to shut down its OSD and keep it down? There is an example to stop the OSD on the server using systemctl, outside of cephadm:* ssh {osd-host}sudo systemctl stop ceph-osd@{osd-num} Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

2 1

RGW cannot list or create openidconnect providers
by mat＠hazmat.dev 17 Feb '23

17 Feb '23

Hello, I'm attempting to setup an OpenIDConnect provider with RGW. I'm doing this using the boto3 API & Python. However it seems that the APIs are failing in some unexpected ways because radosgw was not setup correctly. There is sample code below, and yes, I know there are "secrets" in it - but this is an offline test lab so I am fine with this. The first error shows this in the logs. 2023-02-16T00:45:26.860-0500 7fe19fef7700 1 ====== starting new request req=0x7fe2ccb54680 ===== 2023-02-16T00:45:26.904-0500 7fe19def3700 0 req 17562030806519127926 0.044000439s ERROR: listing filtered objects failed: OIDC pool: default.rgw.meta: oidc_url.: (2) No such file or directory 2023-02-16T00:45:26.904-0500 7fe19aeed700 1 ====== req done req=0x7fe2ccb54680 op status=-2 http_status=404 latency=0.044000439s ====== 2023-02-16T00:45:26.904-0500 7fe19aeed700 1 beast: 0x7fe2ccb54680: 10.20.104.178 - authentik [16/Feb/2023:00:45:26.860 -0500] "POST / HTTP/1.1" 404 189 - "Boto3/1.26.71 Python/3.11.1 Linux/6.0.6-76060006-generic Botocore/1.29.72" - latency=0.044000439s So the object "oidc_url" is missing from the "default.rgw.meta" pool? rados --pool default.rgw.meta ls --all users.uid root.buckets users.uid authentik.buckets root test4 root .bucket.meta.test2:3866fac0-854b-48b5-b3b7-bf84a166a404.1165645.1 users.keys ZVBTLTYRRPY7JU39WOR9 users.uid authentik users.uid cephadmin users.keys NIVIV0JSKD9D2LDC3IH4 users.uid root users.email tester(a)lab.dev users.keys L70QT3LN71SQXWHS97Y4 root .bucket.meta.test:3866fac0-854b-48b5-b3b7-bf84a166a404.1204730.1 root .bucket.meta.test4:3866fac0-854b-48b5-b3b7-bf84a166a404.1204730.2 root test root test2 Well the object is clearly not there and I do not know how to fix this. The second error produces this error in the log: 2023-02-16T01:11:29.304-0500 7fe1976e6700 1 ====== starting new request req=0x7fe2ccb54680 ===== 2023-02-16T01:11:29.312-0500 7fe18c6d0700 1 ====== req done req=0x7fe2ccb54680 op status=-22 http_status=400 latency=0.008000083s ====== 2023-02-16T01:11:29.312-0500 7fe18c6d0700 1 beast: 0x7fe2ccb54680: 10.20.104.178 - authentik [16/Feb/2023:01:11:29.304 -0500] "POST / HTTP/1.1" 400 189 - "Boto3/1.26.71 Python/3.11.1 Linux/6.0.6-76060006-generic Botocore/1.29.72" - latency=0.008000083s Its much less clear what is going on here, it just returns 400. Boto raises this exception, "botocore.exceptions.ClientError: An error occurred (Unknown) when calling the CreateOpenIDConnectProvider operation: Unknown". Has anyone seen this before and know how to setup the correct objects for OpenidConnect? Version info ============================================== ceph version 17.2.5 (e04241aa9b639588fa6c864845287d2824cb6b55) quincy (stable) Examples below ============================================== # creating the client works fine - I can see my user authenticate in the radosgw logs access_key_id = 'L70QT3LN71SQXWHS97Y4' secret_access_key = 'QEXLa5V0Zm38068n3goDtm8V6WlaDwxVmAq9W2XV' iam = boto3.client('iam', aws_access_key_id=access_key_id, aws_secret_access_key=secret_access_key, region_name="default", endpoint_url="https://s3.lab") # First error providers_response = iam.list_open_id_connect_providers() # Second Error oidc_response = iam.create_open_id_connect_provider( # Issuer URL Url="https://login.lab/application/o/d7d64496e26c156ca9ea0802c5d7ed1c/", ClientIDList=['authentik'], ThumbprintList=['BDCC44F40254E7E1258DA4698833FFE2E8AECA3D3799044D8A1F97F7DFF20511'])

2 2

ceph noout vs ceph norebalance, which is better for minor maintenance
by wkonitzer＠mirantis.com 17 Feb '23

17 Feb '23

Hi, We have a discussion going on about which is the correct flag to use for some maintenance on an OSD, should it be "noout" or "norebalance"? This was sparked because we need to take an OSD out of service for a short while to upgrade the firmware. One school of thought is: - "ceph norebalance" prevents automatic rebalancing of data between OSDs, which Ceph does to ensure all OSDs have roughly the same amount of data. - "ceph noout" on the other hand prevents OSDs from being marked as out-of-service during maintenance, which helps maintain cluster performance and availability. - Additionally, if another OSD fails while the "norebalance" flag is set, the data redundancy and fault tolerance of the Ceph cluster may be compromised. - So if we're going to maintain the performance and reliability we need to set the "ceph noout" flag to prevent the OSD from being marked as OOS during maintenance and allow the automatic data redistribution feature of Ceph to work as intended. The other opinion is: - With the noout flag set, Ceph clients are forced to think that OSD exists and is accessible - so they continue sending requests to such OSD. The OSD also remains in the crush map without any signs that it is actually out. If an additional OSD fails in the cluster with the noout flag set, Ceph is forced to continue thinking that this new failed OSD is OK. It leads to stalled or delayed response from the OSD side to clients. - Norebalance instead takes into account the in/out OSD status, but prevents data rebalance. Clients are also aware of the real OSD status, so no requests go to the OSD which is actually out. If an additional OSD fails - only the required temporary PG are created to maintain at least 2 existing copies of the same data (well, generally it is set by the pool min size). The upstream docs seem pretty clear that noout should be used for maintenance (https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-osd/) but the second opinion strongly suggests that norebalance is actually better and the Ceph docs are out of date. So what is the feedback from the wider community? Thanks, Will

5 7

bluefs_db_type
by Stolte, Felix 17 Feb '23

17 Feb '23

Hey guys, most of my osds have HDD for block and SSD for db. But according to "ceph osd metadata" bluefs_db_type = hdd and bluefs_db_rotational = 1. lsblk -o name, rota reveals the following (sdb is db device for 3 hdds): sdb 0 ├─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--b164ba4c--48c9--41a0--8b5e--ae3a6a23a22c 1 ├─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--1c7aa9a1--791d--4ed6--8049--9fba8d5ac828 1 └─ceph--block--dbs--b77a8d7c--bdb5--420b--ad27--65e1d5080550-osd--block--db--ec92f9c6--d651--46ed--b6cd--4cf37c8ce284 1 I am pretty sure sdb had rota 0 during osd deployment (still has, but lvm volumes don’t). Question 1: Is the output of osd metadata for bluefs_db_type and bluefs_db_rotational relevant for how the osd process is treating the disks? (or does it just reflect the value of /sys/block/<device>/queue/rotational?) Question 2: How can I verify that the osd process is treating the db device as an ssd? If i remember correctly, the osd process is using different parameters if db is on SSD instead of HDD. Best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Volker Rieke Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

1 0

RGW Service SSL HAProxy.cfg
by Jimmy Spets 17 Feb '23

17 Feb '23

Hi I am trying to setup the “High availability service for RGW” using SSL both to the HAProxy and from the HAProxy to the RGW backend. The SSL certificate gets applied to both HAProxy and the RGW. If I use the RGW instances directly they work as expected. The RGW config is as follows: service_type: rgw service_id: rgw service_name: rgw.rgw placement: label: rgw count_per_host: 2 spec: ssl: true rgw_frontend_port: 6443 rgw_frontend_ssl_certificate: | -----BEGIN CERTIFICATE---- -----END PRIVATE KEY----- Ingress as follows: service_type: ingress service_id: rgw.rgw placement: hosts: - cephrgw01 - cephrgw02 - cephrgw03 spec: backend_service: rgw.rgw virtual_ip: 172.16.1.130/16 frontend_port: 443 monitor_port: 1967 ssl_cert: | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- The issue is that the haproxy.cfg gets generated like this, without SSL enabled on the backends: # This file is generated by cephadm. global log127.0.0.1 local2 chroot/var/lib/haproxy pidfile/var/lib/haproxy/haproxy.pid maxconn8000 daemon stats socket /var/lib/haproxy/stats defaults modehttp logglobal optionhttplog optiondontlognull option http-server-close option forwardforexcept 127.0.0.0/8 optionredispatch retries3 timeout queue20s timeout connect5s timeout http-request1s timeout http-keep-alive 5s timeout client1s timeout server1s timeout check5s maxconn8000 frontend stats mode http bind 172.16.1.130:1967 bind localhost:1967 stats enable stats uri /stats stats refresh 10s stats auth admin:abcdefg http-request use-service prometheus-exporter if { path /metrics } monitor-uri /health frontend frontend bind 172.16.1.130:443 ssl crt /var/lib/haproxy/haproxy.pem default_backend backend backend backend option forwardfor balance static-rr option httpchk HEAD / HTTP/1.0 server rgw.rgw.cephrgw01.euvqmd 172.16.1.131:6443 check weight 100 server rgw.rgw.cephrgw01.aphsnx 172.16.1.131:6444 check weight 100 server rgw.rgw.cephrgw02.ovckaw 172.16.1.132:6443 check weight 100 server rgw.rgw.cephrgw02.jevtrb 172.16.1.132:6444 check weight 100 server rgw.rgw.cephrgw03.gzdame 172.16.1.133:6443 check weight 100 server rgw.rgw.cephrgw03.bchspq 172.16.1.133:6444 check weight 100 This of course does not work as the backend use SSL. Is there some configuration that I have missed or should I file a bug report? /Jimmy

2 3

ceph-osd@86.service crashed at a random time.
by luckydog xf 17 Feb '23

17 Feb '23

Hello, lists. I have a 108 OSD ceph cluster. All OSDs work fine except one OSD-86. ceph-osd(a)86.service stopped working at a random time. The disk is normal by checking with `smarctl -a`. It could be fine for a few days after I restart it. Then it goes wrong again. I paste the related log here. It stopped at 05:26 UTC. --- 2023-02-17T05:26:37.795+0000 7ff525846700 0 log_channel(cluster) log [DBG] : 17.df scrub starts 2023-02-17T05:26:37.799+0000 7ff525846700 0 log_channel(cluster) log [DBG] : 17.df scrub ok 2023-02-17T05:26:38.779+0000 7ff527049700 0 log_channel(cluster) log [DBG] : 2.64 scrub starts 2023-02-17T05:26:38.783+0000 7ff527049700 0 log_channel(cluster) log [DBG] : 2.64 scrub ok 2023-02-17T05:26:38.871+0000 7ff526848700 1 osd.86 pg_epoch: 113734 pg[20.115( v 113733'56242916 (113711'56240668,113733'56242916] local-lis/les=113726/113727 n=1113 ec=440/440 lis/c=113726/113726 les/c/f=113727/113727/0 sis=113734) [105,86,97] r=1 lpr=113734 pi=[113726,113734)/1 luod=0'0 lua=113730'56242903 crt=113733'56242916 lcod 113733'56242915 mlcod 0'0 active mbc={}] start_peering_interval up [105,86,97] -> [105,86,97], acting [105,97] -> [105,86,97], acting_primary 105 -> 105, up_primary 105 -> 105, role -1 -> 1, features acting 4540138292840890367 upacting 4540138292840890367 2023-02-17T05:26:38.871+0000 7ff526848700 1 osd.86 pg_epoch: 113734 pg[20.115( v 113733'56242916 (113711'56240668,113733'56242916] local-lis/les=113726/113727 n=1113 ec=440/440 lis/c=113726/113726 les/c/f=113727/113727/0 sis=113734) [105,86,97] r=1 lpr=113734 pi=[113726,113734)/1 crt=113733'56242916 lcod 113733'56242915 mlcod 0'0 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray 2023-02-17T05:26:55.075+0000 7ff52784a700 -1 *** Caught signal (Segmentation fault) ** in thread 7ff52784a700 thread_name:tp_osd_tp ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable) 1: (()+0x14420) [0x7ff54448a420] 2: (BlueStore::ExtentMap::decode_some(ceph::buffer::v15_2_0::list&)+0x31d) [0x561eeca36ebd] 3: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x241) [0x561eeca3de21] 4: (BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int, unsigned long)+0x153) [0x561eeca4ae53] 5: (BlueStore::read(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x233) [0x561eeca4bf63] 6: (ReplicatedBackend::be_deep_scrub(hobject_t const&, ScrubMap&, ScrubMapBuilder&, ScrubMap::object&)+0x2b5) [0x561eec873235] 7: (PGBackend::be_scan_list(ScrubMap&, ScrubMapBuilder&)+0x35f) [0x561eec6f2b6f] 8: (PG::build_scrub_map_chunk(ScrubMap&, ScrubMapBuilder&, hobject_t, hobject_t, bool, ThreadPool::TPHandle&)+0x8b) [0x561eec5aa00b] 9: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x14c8) [0x561eec5bc648] 10: (PG::scrub(unsigned int, ThreadPool::TPHandle&)+0x31b) [0x561eec5be67b] 11: (ceph::osd::scheduler::PGScrub::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x16) [0x561eec7876b6] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x4db) [0x561eec51724b] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x403) [0x561eecbd5353] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561eecbd8154] 15: (()+0x8609) [0x7ff54447e609] 16: (clone()+0x43) [0x7ff5443a3133] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -7193> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command assert hook 0x561ef68ea610 -7192> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command abort hook 0x561ef68ea610 -7191> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command leak_some_memory hook 0x561ef68ea610 -7190> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command perfcounters_dump hook 0x561ef68ea610 -7189> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command 1 hook 0x561ef68ea610 -7188> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command perf dump hook 0x561ef68ea610 -7187> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command perfcounters_schema hook 0x561ef68ea610 -7186> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command perf histogram dump hook 0x561ef68ea610 -7185> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command 2 hook 0x561ef68ea610 -7184> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command perf schema hook 0x561ef68ea610 -7183> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command perf histogram schema hook 0x561ef68ea610 -7182> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command perf reset hook 0x561ef68ea610 -7181> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command config show hook 0x561ef68ea610 -7180> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command config help hook 0x561ef68ea610 -7179> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command config set hook 0x561ef68ea610 -7178> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command config unset hook 0x561ef68ea610 -7177> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command config get hook 0x561ef68ea610 -7176> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command config diff hook 0x561ef68ea610 -7175> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command config diff get hook 0x561ef68ea610 -7174> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command injectargs hook 0x561ef68ea610 -7173> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command log flush hook 0x561ef68ea610 -7172> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command log dump hook 0x561ef68ea610 -7171> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command log reopen hook 0x561ef68ea610 -7170> 2023-02-17T05:26:23.928+0000 7ff5440ded80 5 asok(0x561ef6990000) register_command dump_mempools hook 0x561ef7568068 -7169> 2023-02-17T05:26:23.936+0000 7ff5440ded80 10 monclient: get_monmap_and_config -7168> 2023-02-17T05:26:23.936+0000 7ff5440ded80 10 monclient: build_initial_monmap -7167> 2023-02-17T05:26:23.936+0000 7ff5440ded80 10 monclient: monmap: epoch 0 -- -50> 2023-02-17T05:26:44.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:14.771302+0000) -49> 2023-02-17T05:26:44.771+0000 7ff52103d700 5 osd.86 113735 heartbeat osd_stat(store_statfs(0x35e3cac2000/0x40000000/0x3aac7ffe000, data 0x45247e7e86/0x450493c000, compress 0x0/0x0/0x0, omap 0x155e40, meta 0x3feaa1c0), peers [0,9,11,12,13,15,17,18,19,21,22,24,26,27,30,31,33,34,37,39,40,43,46,49,51,55,56,57,60,61,62,64,65,66,67,69,70,71,73,78,79,80,82,83,84,85,87,88,89,91,92,93,94,96,97,100,101,102,103,105,106,107] op hist []) -48> 2023-02-17T05:26:45.179+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 550584320 unmapped: 1384448 heap: 551968768 old mem: 2845415832 new mem: 2845415832 -47> 2023-02-17T05:26:45.447+0000 7ff5430b4700 10 monclient: handle_auth_request added challenge on 0x561f1606f000 -46> 2023-02-17T05:26:45.767+0000 7ff535866700 10 monclient: tick -45> 2023-02-17T05:26:45.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:15.771503+0000) -44> 2023-02-17T05:26:46.183+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 550805504 unmapped: 1163264 heap: 551968768 old mem: 2845415832 new mem: 2845415832 -43> 2023-02-17T05:26:46.579+0000 7ff5428b3700 10 monclient: handle_auth_request added challenge on 0x561f1606ec00 -42> 2023-02-17T05:26:46.579+0000 7ff536868700 2 osd.86 113735 ms_handle_reset con 0x561f1606ec00 session 0x561f166c6f00 -41> 2023-02-17T05:26:46.767+0000 7ff535866700 10 monclient: tick -40> 2023-02-17T05:26:46.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:16.771672+0000) -39> 2023-02-17T05:26:47.075+0000 7ff52103d700 5 osd.86 113735 heartbeat osd_stat(store_statfs(0x35e3cac2000/0x40000000/0x3aac7ffe000, data 0x45247e7e86/0x450493c000, compress 0x0/0x0/0x0, omap 0x155e40, meta 0x3feaa1c0), peers [0,9,11,12,13,15,17,18,19,21,22,24,26,27,30,31,33,34,37,39,40,43,46,49,51,55,56,57,60,61,62,64,65,66,67,69,70,71,73,78,79,80,82,83,84,85,87,88,89,91,92,93,94,96,97,100,101,102,103,105,106,107] op hist []) -38> 2023-02-17T05:26:47.183+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 553959424 unmapped: 1155072 heap: 555114496 old mem: 2845415832 new mem: 2845415832 -37> 2023-02-17T05:26:47.183+0000 7ff537069700 5 bluestore.MempoolThread(0x561ef7616a68) _resize_shards cache_size: 2845415832 kv_alloc: 1140850688 kv_used: 110273280 meta_alloc: 1023410176 meta_used: 2977563 data_alloc: 654311424 data_used: 0 -36> 2023-02-17T05:26:47.575+0000 7ff52103d700 5 osd.86 113735 heartbeat osd_stat(store_statfs(0x35e3cac2000/0x40000000/0x3aac7ffe000, data 0x45247e7e86/0x450493c000, compress 0x0/0x0/0x0, omap 0x155e40, meta 0x3feaa1c0), peers [0,9,11,12,13,15,17,18,19,21,22,24,26,27,30,31,33,34,37,39,40,43,46,49,51,55,56,57,60,61,62,64,65,66,67,69,70,71,73,78,79,80,82,83,84,85,87,88,89,91,92,93,94,96,97,100,101,102,103,105,106,107] op hist []) -35> 2023-02-17T05:26:47.767+0000 7ff535866700 10 monclient: tick -34> 2023-02-17T05:26:47.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:17.771928+0000) -33> 2023-02-17T05:26:48.223+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 557449216 unmapped: 811008 heap: 558260224 old mem: 2845415832 new mem: 2845415832 -32> 2023-02-17T05:26:48.699+0000 7ff5428b3700 10 monclient: handle_auth_request added challenge on 0x561efca9f000 -31> 2023-02-17T05:26:48.767+0000 7ff535866700 10 monclient: tick -30> 2023-02-17T05:26:48.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:18.772104+0000) -29> 2023-02-17T05:26:49.227+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 560799744 unmapped: 606208 heap: 561405952 old mem: 2845415832 new mem: 2845415832 -28> 2023-02-17T05:26:49.275+0000 7ff52103d700 5 osd.86 113735 heartbeat osd_stat(store_statfs(0x35e3cac0000/0x40000000/0x3aac7ffe000, data 0x45247e7e86/0x450493c000, compress 0x0/0x0/0x0, omap 0x155e40, meta 0x3feaa1c0), peers [0,9,11,12,13,15,17,18,19,21,22,24,26,27,30,31,33,34,37,39,40,43,46,49,51,55,56,57,60,61,62,64,65,66,67,69,70,71,73,78,79,80,82,83,84,85,87,88,89,91,92,93,94,96,97,100,101,102,103,105,106,107] op hist []) -27> 2023-02-17T05:26:49.367+0000 7ff5438b5700 10 monclient: handle_auth_request added challenge on 0x561f13c69000 -26> 2023-02-17T05:26:49.767+0000 7ff535866700 10 monclient: tick -25> 2023-02-17T05:26:49.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:19.772303+0000) -24> 2023-02-17T05:26:50.231+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 565821440 unmapped: 827392 heap: 566648832 old mem: 2845415832 new mem: 2845415832 -23> 2023-02-17T05:26:50.295+0000 7ff5430b4700 10 monclient: handle_auth_request added challenge on 0x561efca9f400 -22> 2023-02-17T05:26:50.767+0000 7ff535866700 10 monclient: tick -21> 2023-02-17T05:26:50.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:20.772449+0000) -20> 2023-02-17T05:26:51.231+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 570171392 unmapped: 671744 heap: 570843136 old mem: 2845415832 new mem: 2845415832 -19> 2023-02-17T05:26:51.767+0000 7ff535866700 10 monclient: tick -18> 2023-02-17T05:26:51.767+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:21.772614+0000) -17> 2023-02-17T05:26:51.803+0000 7ff5428b3700 10 monclient: handle_auth_request added challenge on 0x561f13c68400 -16> 2023-02-17T05:26:52.123+0000 7ff53205f700 5 bluestore(/var/lib/ceph/osd/ceph-86) _kv_sync_thread utilization: idle 9.937826090s of 10.006035168s, submitted: 179 -15> 2023-02-17T05:26:52.183+0000 7ff537069700 5 bluestore.MempoolThread(0x561ef7616a68) _resize_shards cache_size: 2845415832 kv_alloc: 1140850688 kv_used: 113276944 meta_alloc: 1040187392 meta_used: 16149983 data_alloc: 654311424 data_used: 0 -14> 2023-02-17T05:26:52.247+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 574758912 unmapped: 278528 heap: 575037440 old mem: 2845415832 new mem: 2845415832 -13> 2023-02-17T05:26:52.343+0000 7ff5438b5700 10 monclient: handle_auth_request added challenge on 0x561f13c68000 -12> 2023-02-17T05:26:52.539+0000 7ff5430b4700 10 monclient: handle_auth_request added challenge on 0x561f167b4800 -11> 2023-02-17T05:26:52.555+0000 7ff5428b3700 10 monclient: handle_auth_request added challenge on 0x561f14eb4800 -10> 2023-02-17T05:26:52.771+0000 7ff535866700 10 monclient: tick -9> 2023-02-17T05:26:52.771+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:22.772842+0000) -8> 2023-02-17T05:26:52.775+0000 7ff52103d700 5 osd.86 113735 heartbeat osd_stat(store_statfs(0x35e3cabe000/0x40000000/0x3aac7ffe000, data 0x45247e7e86/0x450493c000, compress 0x0/0x0/0x0, omap 0x155e40, meta 0x3feaa1c0), peers [0,9,11,12,13,15,17,18,19,21,22,24,26,27,30,31,33,34,37,39,40,43,46,49,51,55,56,57,60,61,62,64,65,66,67,69,70,71,73,78,79,80,82,83,84,85,87,88,89,91,92,93,94,96,97,100,101,102,103,105,106,107] op hist []) -7> 2023-02-17T05:26:53.247+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 579764224 unmapped: 516096 heap: 580280320 old mem: 2845415832 new mem: 2845415832 -6> 2023-02-17T05:26:53.531+0000 7ff5438b5700 10 monclient: handle_auth_request added challenge on 0x561f14eb4000 -5> 2023-02-17T05:26:53.771+0000 7ff535866700 10 monclient: tick -4> 2023-02-17T05:26:53.771+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:23.773042+0000) -3> 2023-02-17T05:26:54.251+0000 7ff537069700 5 prioritycache tune_memory target: 4294967296 mapped: 583467008 unmapped: 1007616 heap: 584474624 old mem: 2845415832 new mem: 2845415832 -2> 2023-02-17T05:26:54.771+0000 7ff535866700 10 monclient: tick -1> 2023-02-17T05:26:54.771+0000 7ff535866700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-02-17T05:26:24.773241+0000) 0> 2023-02-17T05:26:55.075+0000 7ff52784a700 -1 *** Caught signal (Segmentation fault) ** in thread 7ff52784a700 thread_name:tp_osd_tp ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable) 1: (()+0x14420) [0x7ff54448a420] 2: (BlueStore::ExtentMap::decode_some(ceph::buffer::v15_2_0::list&)+0x31d) [0x561eeca36ebd] 3: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x241) [0x561eeca3de21] 4: (BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int, unsigned long)+0x153) [0x561eeca4ae53] 5: (BlueStore::read(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x233) [0x561eeca4bf63] 6: (ReplicatedBackend::be_deep_scrub(hobject_t const&, ScrubMap&, ScrubMapBuilder&, ScrubMap::object&)+0x2b5) [0x561eec873235] 7: (PGBackend::be_scan_list(ScrubMap&, ScrubMapBuilder&)+0x35f) [0x561eec6f2b6f] 8: (PG::build_scrub_map_chunk(ScrubMap&, ScrubMapBuilder&, hobject_t, hobject_t, bool, ThreadPool::TPHandle&)+0x8b) [0x561eec5aa00b] 9: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x14c8) [0x561eec5bc648] 10: (PG::scrub(unsigned int, ThreadPool::TPHandle&)+0x31b) [0x561eec5be67b] 11: (ceph::osd::scheduler::PGScrub::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x16) [0x561eec7876b6] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x4db) [0x561eec51724b] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x403) [0x561eecbd5353] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561eecbd8154] 15: (()+0x8609) [0x7ff54447e609] 16: (clone()+0x43) [0x7ff5443a3133] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_rwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore

1 0

forever stuck "slow ops" osd
by Arvid Picciani 16 Feb '23

16 Feb '23

Hi, today our entire cluster froze. or anything that uses librbd to be specific. ceph version 16.2.10 The message that saved me was "256 slow ops, oldest one blocked for 2893 sec, osd.7 has slow ops" , because it makes it immediately clear that this osd is the issue. I stopped the osd, which made the cluster available again. Restarting the osd makes it stuck again, although that osd has nothing in the error log, and the underlying ssd is healthy. It's just that one out of 27. There's nothing unique about it. We use the same disk product in other osds, and the host is also running other osds just fine. How does this happen, and why can the cluster not recover from this automatically? For example by stopping the affected osd or at least having a timeout for ops. Thanks -- +4916093821054

2 1

clt meeting summary [15/02/2023]
by Nizamudeen A 16 Feb '23

16 Feb '23

Hi all, today's topics were: - Labs: - Keeping a catalog - Have a dedicated group to debug/work through the issues. - Looking for interested parties that would like to contribute in the lab maintenance tasks - Poll for meeting time, looking for a central person to follow up / organize - No one's been actively coordinating on the lab issues apart from Laura. David Orman volunteered if we need help coordinating the lab issues - Reef release - [casey] things aren't looking good for end-of-february freeze - Since the whole thing depends on test-infra, can't really estimate the time frame. - The freeze maybe delayed - Dev Summit in Amsterdam: estimate how many would attend in person, remote - 50/50 of those present would attend (as per the voting) - Ad hoc virtual could work - Need to update the component leads page: https://ceph.io/en/community/team/ - Vikhyath volunteered before, so Josh will check with him. Regards, -- Nizamudeen A Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

2 2

User + Dev monthly meeting happening tomorrow, Feb. 16th!
by Laura Flores 16 Feb '23

16 Feb '23

Hi Ceph Users, The User + Dev monthly meeting is coming up tomorrow, Thursday, Feb. 16th at 3:00 PM UTC. Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes <https://pad.ceph.com/p/ceph-user-dev-monthly-minutes#L13> See you there, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage Red Hat Inc. <https://www.redhat.com> Chicago, IL lflores(a)redhat.com M: +17087388804 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 1

RGW archive zone lifecycle
by ondrej＠kuuk.la 16 Feb '23

16 Feb '23

Hi, I have two Ceph clusters in a multi-zone setup. The first one (master zone) would be accessible to users for their interaction using RGW. The second one is set to sync from the master zone with the tier type of the zone set as an archive (to version all files). My question here is. Is there an option to set a lifecycle for the version files saved on the archive zone? For example, keep only 5 versions per file or delete version files older than one year? Thanks a lot.

2 1

how to sync data on two site CephFS
by zxcs 16 Feb '23

16 Feb '23

Hi, Experts, we already have a CephFS cluster, called A, and now we want to setup another CephFS cluster(called B) in other site. And we need to synchronize data with each other for some directory(if all directory can synchronize , then very very good), Means when we write a file in A cluster, this file can auto sync to B cluster, and when we create a file or directory on B Cluster, this file or directory can auto sync to A Cluster. our question is does there any best practices to do that on CephFS? Thanks in advance! Thanks, zx

3 2

Extremally need help. Openshift cluster is down :c
by kreept.sama＠gmail.com 16 Feb '23

16 Feb '23

Hello everyone and sorry. Maybe someone has already faced this problem. A day ago, we restored our Openshift cluster, however, at the moment, the PVCs cannot connect to the pod. We looked at the status of the ceph and found that our MDS were in standby mode, then found that the metadata was corrupted. After some manipulations, we were able to turn on our MDS daemons, but there is still no record on the cluster, the ceph status command shows the following. sh-4.4$ ceph -s cluster: id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119 health: HEALTH_ERR 1 MDSs report damaged metadata 1 MDSs are read only 6 daemons have recently crashed services: mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 26h) mgr: a(active, since 25h) mds: 1/1 daemons up, 1 hot standby osd: 9 osds: 9 up (since 41h), 9 in (since 42h) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 225 pgs objects: 1.60M objects, 234 GiB usage: 606 GiB used, 594 GiB / 1.2 TiB avail pgs: 225 active+clean io: client: 852 B/s rd, 1 op/s rd, 0 op/s wr Now we trying to follow this instructions: https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#recovery-… What else have we tried: cephfs-journal-tool --rank=1:0 event recover_dentries summary cephfs-journal-tool --rank=1:0 journal reset cephfs-table-tool all reset session ceph tell mds.gml--cephfs-a scrub start / recursive repair force ceph tell mds.gml--cephfs-b scrub start / recursive repair force ceph mds repaired 0 ceph tell mds.gml--cephfs-a damage ls [ { "damage_type": "dir_frag", "id": 26851730, "ino": 1100162409473, "frag": "*", "path": "/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027796_1027796_0" }, ……… { "damage_type": "dir_frag", "id": 118336643, "ino": 1100162424469, "frag": "*", "path": "/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027832_1027832_0" }, Now we trying: # Session table cephfs-table-tool 0 reset session # SnapServer cephfs-table-tool 0 reset snap # InoTable cephfs-table-tool 0 reset inode # Journal cephfs-journal-tool --rank=0 journal reset # Root inodes ("/" and MDS directory) cephfs-data-scan init cephfs-data-scan scan_extents <data pool> cephfs-data-scan scan_inodes <data pool> cephfs-data-scan scan_links Is it right way and cant it be our salvation? Thank you!

2 10

Renaming a ceph node
by Rice, Christian 16 Feb '23

16 Feb '23

Can anyone please point me at a doc that explains the most efficient procedure to rename a ceph node WITHOUT causing a massive misplaced objects churn? When my node came up with a new name, it properly joined the cluster and owned the OSDs, but the original node with no devices remained. I expect this affected the crush map such that a large qty of objects got reshuffled. I want no object movement, if possible. BTW this old cluster is on luminous. ☹

3 4

Ceph (cepadm) quincy: can't add osd from remote nodes.
by Anton Chivkunov 16 Feb '23

16 Feb '23

Hello! I stuck with a problem, while trying to create cluster of 3 nodes (AWS EC2 instancies): fa11 ~ # ceph orch host ls HOST ADDR LABELS STATUS fa11 172.16.24.67 _admin fa12 172.16.23.159 _admin fa13 172.16.25.119 _admin 3 hosts in cluster Each of them have 2 disks (all accepted by CEPH): fa11 ~ # ceph orch device ls HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS fa11 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol016651cf7f3b9c9dd 8589M Yes 7m ago fa11 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol034082d7d364dfbdb 5368M Yes 7m ago fa12 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol0ec193fa3f77fee66 8589M Yes 3m ago fa12 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol018736f7eeab725f5 5368M Yes 3m ago fa13 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol0443a031550be1024 8589M Yes 84s ago fa13 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol0870412d37717dc2c 5368M Yes 84s ago fa11 is first host, where from I manage cluster. Adding OSD from fa11 itself works fine: fa11 ~ # ceph orch daemon add osd fa11:/dev/nvme1n1 Created osd(s) 0 on host 'fa11' But it doesn't work for other 2 hosts (it hangs forever): fa11 ~ # ceph orch daemon add osd fa12:/dev/nvme1n1 ^CInterrupted Logs on fa12 shows that it hangs at following step: fa12 ~ # tail /var/log/ceph/a9ef6c26-ac38-11ed-9429-06e6bc29c1db/ceph-volume.log ... [2023-02-14 07:38:20,942][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key [2023-02-14 07:38:20,964][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a51506c2-e910-4763-9a0c-f6c2194944e2 I'm not sure what might be the reason for this hanging? *Additional details: 1) cephadm installed, using curl (https://docs.ceph.com/en/quincy/cephadm/install/#curl-based-installation) 2) I use user "ceph", instead of "root" and port 2222 instead of 22. First node was bootstrapped, using below command: cephadm bootstrap --mon-ip 172.16.24.67 --allow-fqdn-hostname --ssh-user ceph --ssh-config /home/anton/ceph/ssh_config --cluster-network 172.16.16.0/20 --skip-monitoring-stack Content of /home/anton/ceph/ssh_config: fa11 ~ # cat /home/anton/ceph/ssh_config Host * User ceph Port 2222 IdentityFile /home/ceph/.ssh/id_rsa StrictHostKeyChecking no UserKnownHostsFile=/dev/null 3) Hosts fa12 and fa13 were added, using commnds: ceph orch host add fa12.testing.swiftserve.com 172.16.23.159 --labels _admin ceph orch host add fa13.testing.swiftserve.com 172.16.25.119 --labels _admin Thanks in advance! BR/Anton

3 3

Swift Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True
by Beaman, Joshua 15 Feb '23

15 Feb '23

Greetings, Regarding: https://tracker.ceph.com/issues/58019 And: https://github.com/ceph/ceph/pull/47341 We have completed upgrading our production pacific clusters to 16.2.11 and are still experiencing this bug. Can anyone confirm if this backport was included in 16.2.11. Could there be anything wrong with our configuration that is contributing? Thank you, Josh Beaman

1 0

Missing object in bucket list
by mahnoosh shahidi 15 Feb '23

15 Feb '23

Hi all, We have a cluster on 15.2.12. We are experiencing an unusual scenario in S3. User send PUT request to upload an object and RGW returns 200 as a response status code. The object has been uploaded and can be downloaded but it does not exist in the bucket list. We also tried to get the bucket index entry for that object but it does not exist. Below is the log of the RGW for the request. 1 ====== starting new request req=0x7f246c4426b0 ===== > 2 req 44161 0s initializing for trans_id = > tx00000000000000000ac81-0063e36653-17e18f0-default > 10 rgw api priority: s3=3 s3website=2 > 10 host=192.168.0.201 > 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 > 10 meta>> HTTP_X_AMZ_DATE > 10 x>> > x-amz-content-sha256:30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > 10 x>> x-amz-date:20230208T090731Z > 10 handler=22RGWHandler_REST_Obj_S3 > 2 req 44161 0s getting op 1 > 10 req 44161 0s s3:put_obj scheduling with dmclock client=2 cost=1 > 10 op=21RGWPutObj_ObjStore_S3 > 2 req 44161 0s s3:put_obj verifying requester > 10 v4 signature format = > 7daee7e343e08d8121e843c6c77da3cc827bd4f4f179548e1c729c130a3e7745 > 10 v4 credential format = > 85ZYESW8HS34DC95MZBT/20230208/us-east-1/s3/aws4_request > 10 access key id = 85ZYESW8HS34DC95MZBT > 10 credential scope = 20230208/us-east-1/s3/aws4_request > 10 req 44161 0s canonical headers format = > content-md5:ttgbNgpWctgMJ0MPORU+LA== > host:192.168.0.201 > > x-amz-content-sha256:30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > x-amz-date:20230208T090731Z > > 10 payload request hash = > 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > 10 canonical request = PUT > /test7/file508294 > > content-md5:ttgbNgpWctgMJ0MPORU+LA== > host:192.168.0.201 > > x-amz-content-sha256:30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > x-amz-date:20230208T090731Z > > content-md5;host;x-amz-content-sha256;x-amz-date > 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > 10 canonical request hash = > 2ab4fe4f0fa402435c3237382bdc77e86203406e90a2768a70410f58754bb6ba > 10 string to sign = AWS4-HMAC-SHA256 > 20230208T090731Z > 20230208/us-east-1/s3/aws4_request > 2ab4fe4f0fa402435c3237382bdc77e86203406e90a2768a70410f58754bb6ba > 10 req 44161 0s delaying v4 auth > 10 date_k = > a9dc6afa32600995d313f1b6a4fa40be3a3cd574d25db8789ac966a8e7f43356 > 10 region_k = > b9193e8e261f702b88549da7e81e6a4a7672725996ea8a86269fed665b39670d > 10 service_k = > 34214c91aec1192bcc413e02044e346b31ed4f13df8c15830bdb1d7bd3565126 > 10 signing_k = > 7656d62334d92c982f8c21e0200e760054b214eebab6dbeab577fb655c00a6f4 > 10 generated signature = > 7daee7e343e08d8121e843c6c77da3cc827bd4f4f179548e1c729c130a3e7745 > 2 req 44161 0s s3:put_obj normalizing buckets and tenants > 10 s->object=file508294 s->bucket=test7 > 2 req 44161 0s s3:put_obj init permissions > 10 cache get: name=default.rgw.meta+root+test7 : expiry miss > 10 cache put: name=default.rgw.meta+root+test7 info.flags=0x16 > 10 adding default.rgw.meta+root+test7 to cache LRU end > 10 updating xattr: name=ceph.objclass.version bl.length()=42 > 10 cache get: name=default.rgw.meta+root+test7 : type miss > (requested=0x11, cached=0x16) > 10 cache put: name=default.rgw.meta+root+test7 info.flags=0x11 > 10 moving default.rgw.meta+root+test7 to cache LRU end > 10 cache get: name=default.rgw.meta+users.uid+storage : hit > (requested=0x6, cached=0x17) > 10 cache get: name=default.rgw.meta+users.uid+storage : hit > (requested=0x3, cached=0x17) > 2 req 44161 0.003999945s s3:put_obj recalculating target > 2 req 44161 0.003999945s s3:put_obj reading permissions > 2 req 44161 0.003999945s s3:put_obj init op > 2 req 44161 0.003999945s s3:put_obj verifying op mask > 2 req 44161 0.003999945s s3:put_obj verifying op permissions > 5 req 44161 0.003999945s s3:put_obj Searching permissions for > identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=storage, acct_name=storage, subuser=, > perm_mask=15, is_admin=0) mask=50 > 5 Searching permissions for uid=storage > 5 Found permission: 15 > 5 Searching permissions for group=1 mask=50 > 5 Permissions for group not found > 5 Searching permissions for group=2 mask=50 > 5 Permissions for group not found > 5 req 44161 0.003999945s s3:put_obj -- Getting permissions done for > identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=storage, acct_name=storage, subuser=, > perm_mask=15, is_admin=0), owner=storage, perm=2 > 37:31.066+0330 7f2479c5e700 10 req 44161 0.003999945s s3:put_obj > identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=storage, acct_name=storage, subuser=, > perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, > user_perm_mask=2, acl perm=2 > 2 req 44161 0.003999945s s3:put_obj verifying op params > 2 req 44161 0.003999945s s3:put_obj pre-executing > 2 req 44161 0.003999945s s3:put_obj executing > 5 req 44161 0.023999668s s3:put_obj NOTICE: call to > do_aws4_auth_completion > 10 req 44161 0.023999668s s3:put_obj v4 auth ok -- do_aws4_auth_completion > 5 req 44161 0.023999668s s3:put_obj NOTICE: call to > do_aws4_auth_completion > -- > 0 RGWReshardLock::lock failed to acquire lock on > test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 ret=-16 > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 RGWReshardLock::lock failed to acquire lock on > test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 ret=-16 > 10 cache get: name=default.rgw.meta+root+test7 : hit (requested=0x6, > cached=0x17) > 10 cache get: name=default.rgw.meta+root+test7 : hit (requested=0x1, > cached=0x17) > -1 WARNING: The bucket info cache is inconsistent. This is a failure that > should be debugged. I am a nice machine, so I will try to recover. > 10 cache get: > name=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > : hit (requested=0x16, cached=0x17) > 10 cache get: > name=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > : hit (requested=0x13, cached=0x17) > 10 cache put: > name=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > info.flags=0x13 > 10 moving > default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > to cache LRU end > 10 updating xattr: name=ceph.objclass.version bl.length()=42 > 10 updating xattr: name=user.rgw.acl bl.length()=147 > 10 chain_cache_entry: > cache_locator=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > -1 WARNING: The OSD has the same version I have. Something may have gone > squirrelly. An administrator may have forced a change; otherwise there is a > problem somewhere. > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 10 manifest: total_size = 1048576 > 10 setting object > write_tag=c3b354de-c79f-444b-a647-5b272f8148d7.25041136.44161 > 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test7 : hit > (negative entry) > 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test7 : hit > (negative entry) > 10 chain_cache_entry: cache_locator= > 10 cache get: > name=default.rgw.log++pubsub.user.storage.bucket.test7/c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > : hit (negative entry) > 2 req 44161 363.150981592s s3:put_obj completing > 4 write_data failed: Connection reset by peer > 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Connection reset > by peer > 2 req 44161 363.150981592s s3:put_obj op status=0 > 2 req 44161 363.150981592s s3:put_obj http status=200 > 1 ====== req done req=0x7f246c4426b0 op status=0 http_status=200 > latency=363.150981592s ====== Anybody has any idea about the reason for this behaviour? Best Regards, Mahnoosh

2 2

Re: iDRAC 9 version 6.10 shows 0% for write endurance on non-dell drives, work around?
by Konstantin Shalygin 15 Feb '23

15 Feb '23

Hi, You can use smartctl_exporter [1] for all your media, not only the SSD k [1] https://github.com/prometheus-community/smartctl_exporter Sent from my iPhone > On 14 Feb 2023, at 23:01, Drew Weaver <drew.weaver(a)thenap.com> wrote: > Hello, > > After upgrading a lot of iDRAC9 modules to version 6.10 in servers that are involved in a Ceph cluster we noticed that the iDRAC9 shows the write endurance as 0% on any non-certified disk. > > OMSA still shows the correct remaining write endurance but I am assuming that they are working feverishly to eliminate that too. > > I opened a support ticket with Dell once this was brought to my attention and they basically told me that I was lucky that it ever worked at all, which I thought was an odd response given that the iDRAC enterprise licenses cost several hundred dollars each. > > I know that the old Intel Datacenter Tool used to be able to reach through a MegaRAID controller and read the remaining write endurance but that tool is essentially defunct now. > > What are you folks using to monitor your write endurance on your SSDs that you couldn't buy from Dell because they had a 16 week lead time while the MFG could deliver the drives in 3 days? > > Thanks, > -Drew > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Announcing go-ceph v0.17.0
by Sven Anderson 14 Feb '23

14 Feb '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.20.0 Changes include additions to the rbd, rgw and cephfs packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

iDRAC 9 version 6.10 shows 0% for write endurance on non-dell drives, work around?
by Drew Weaver 14 Feb '23

14 Feb '23

Hello, After upgrading a lot of iDRAC9 modules to version 6.10 in servers that are involved in a Ceph cluster we noticed that the iDRAC9 shows the write endurance as 0% on any non-certified disk. OMSA still shows the correct remaining write endurance but I am assuming that they are working feverishly to eliminate that too. I opened a support ticket with Dell once this was brought to my attention and they basically told me that I was lucky that it ever worked at all, which I thought was an odd response given that the iDRAC enterprise licenses cost several hundred dollars each. I know that the old Intel Datacenter Tool used to be able to reach through a MegaRAID controller and read the remaining write endurance but that tool is essentially defunct now. What are you folks using to monitor your write endurance on your SSDs that you couldn't buy from Dell because they had a 16 week lead time while the MFG could deliver the drives in 3 days? Thanks, -Drew

2 2

Cephalocon 2023 Amsterdam Call For Proposals Extended to February 19!
by Mike Perez 14 Feb '23

14 Feb '23

CALL FOR PROPOSALS EXTENDED‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ [image: Logo] CALL FOR PROPOSALS EXTENDED <https://events.linuxfoundation.org/cephalocon/program/cfp/> [image: Ceph23-LaunchAssets-v1_snackable-1200x627-A.png] <https://events.linuxfoundation.org/cephalocon/> We heard your feedback on the short notice for the Call for Proposals <https://events.linuxfoundation.org/cephalocon/program/cfp/> and have extended the deadline to Sunday, February 19, at 11:59 pm PST! Cephalocon <https://events.linuxfoundation.org/cephalocon/> is the premier yearly event that brings together the global community of operators, developers, and researchers for Ceph, the open source distributed storage system designed to provide excellent performance, reliability, and scalability. Join new and existing community members worldwide to learn more about Ceph and the project’s future from the developers writing the code and the operators deploying it at scale. SUBMIT YOUR PROPOSAL <https://events.linuxfoundation.org/cephalocon/program/cfp/> Important Dates - *CFP Closes: *Sunday, February 19 at 11:59 pm PST - *CFP Notifications:* Friday, February 24 - *Schedule Announcement:* Wednesday, March 1 - *Presentation Slide Due Date:* Wednesday, April 12 - *Event Dates:* Monday, April 17 – Tuesday, April 18 (Developer Summit Sunday, April 16) Sponsorship Sponsoring Cephalocon <https://events.linuxfoundation.org/cephalocon/sponsor/> is a unique opportunity to gain valuable mindshare with an elite audience of engineers, researchers, and end-users. Building on the success of Ceph Days and virtual Developer Summits, Cephalocon brings together more than 400 attendees from across the globe to showcase Ceph’s history and its future, real-world applications, and of course, highlight vendor solutions. Cephalocon 2023 promises to make for incredible community building, cross-company collaboration, and cutting-edge training. Contact us at sponsorships(a)ceph.foundation to secure your sponsorship, request additional details or discuss custom options. See Sponsorship Prospectus <https://events.linuxfoundation.org/cephalocon/sponsor/> [image: Link icon] <https://ceph.io> [image: Twitter icon] <https://twitter.com/ceph> [image: YouTube icon] <https://youtube.com/c/cephstorage> [image: LinkedIn icon] <https://www.linkedin.com/company/ceph/> [image: Facebook icon] <https://www.facebook.com/cephstorage/> f8xj3pa8x9xf

2 1

PSA: Potential problems in a recent kernel?
by Matthew Booth 14 Feb '23

14 Feb '23

My ceph cluster became unstable yesterday after zincati (CoreOS's auto-updater) updated one of my nodes from 37.20221225.3.0 to 37.20230110.3.1(*). The symptom was slow ops in my cephfs mds which started immediately the OSDs on this node became in and up. Excluding the OSDs on this node worked round the problem. Note that the node is also running a mon and client workloads which use ceph. Also note that the OSD came up and (IIUC) were participating in recovering their data to other OSDs. The problem only started when I allowed them to be in. I rolled back the OS update and the problem was immediately resolved. Unfortunately I didn't keep the OSD logs, but they lead me to this thread from ceph-users: https://www.mail-archive.com/ceph-users@ceph.io/msg18474.html . I wonder if we have an issue with a very recent kernel update. I should be able to reproduce if it's likely to be of use to anybody, but for now I've rolled back this OS update and disabled automatic updating on my other nodes. Matt (*) The complete list of changes: $ rpm-ostree db diff d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05 fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161 ostree diff commit from: d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05 ostree diff commit to: fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161 Upgraded: bash 5.2.9-3.fc37 -> 5.2.15-1.fc37 btrfs-progs 6.0.2-1.fc37 -> 6.1.2-1.fc37 clevis 18-12.fc37 -> 18-14.fc37 clevis-dracut 18-12.fc37 -> 18-14.fc37 clevis-luks 18-12.fc37 -> 18-14.fc37 clevis-systemd 18-12.fc37 -> 18-14.fc37 container-selinux 2:2.193.0-1.fc37 -> 2:2.198.0-1.fc37 containerd 1.6.12-1.fc37 -> 1.6.14-2.fc37 containers-common 4:1-73.fc37 -> 4:1-76.fc37 containers-common-extra 4:1-73.fc37 -> 4:1-76.fc37 coreutils 9.1-6.fc37 -> 9.1-7.fc37 coreutils-common 9.1-6.fc37 -> 9.1-7.fc37 crun 1.7.2-2.fc37 -> 1.7.2-3.fc37 curl 7.85.0-4.fc37 -> 7.85.0-5.fc37 dnsmasq 2.87-3.fc37 -> 2.88-1.fc37 ethtool 2:6.0-1.fc37 -> 2:6.1-1.fc37 fwupd 1.8.8-1.fc37 -> 1.8.9-1.fc37 git-core 2.38.1-1.fc37 -> 2.39.0-1.fc37 grub2-common 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-efi-x64 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-pc 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-pc-modules 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-tools 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-tools-minimal 1:2.06-63.fc37 -> 1:2.06-72.fc37 kernel 6.0.15-300.fc37 -> 6.0.18-300.fc37 kernel-core 6.0.15-300.fc37 -> 6.0.18-300.fc37 kernel-modules 6.0.15-300.fc37 -> 6.0.18-300.fc37 libcurl-minimal 7.85.0-4.fc37 -> 7.85.0-5.fc37 libgpg-error 1.45-2.fc37 -> 1.46-1.fc37 libgusb 0.4.2-1.fc37 -> 0.4.3-1.fc37 libksba 1.6.2-1.fc37 -> 1.6.3-1.fc37 libpcap 14:1.10.1-4.fc37 -> 14:1.10.2-1.fc37 libpwquality 1.4.4-11.fc37 -> 1.4.5-1.fc37 libsmbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 libwbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 moby-engine 20.10.20-1.fc37 -> 20.10.21-1.fc37 ncurses 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 ncurses-base 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 ncurses-libs 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 net-tools 2.0-0.63.20160912git.fc37 -> 2.0-0.64.20160912git.fc37 rpm-ostree 2022.16-2.fc37 -> 2022.19-2.fc37 rpm-ostree-libs 2022.16-2.fc37 -> 2022.19-2.fc37 samba-client-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 samba-common 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 samba-common-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 selinux-policy 37.16-1.fc37 -> 37.17-1.fc37 selinux-policy-targeted 37.16-1.fc37 -> 37.17-1.fc37 tpm2-tss 3.2.0-3.fc37 -> 3.2.1-1.fc37 Removed: cracklib-dicts-2.9.7-30.fc37.x86_64 -- Matthew Booth

2 1

Does cephfs subvolume have commands similar to `rbd perf` to query iops, bandwidth, and latency of rbd image?
by 郑亮 14 Feb '23

14 Feb '23

Hi all, Does cephfs subvolume have commands similar to rbd perf to query iops, bandwidth, and latency of rbd image? `ceph fs perf stats` shows metrics of the client side, not the metrics of the cephfs subvolume. What I want to get is the metrics at the subvolume level like below. [root@smd-exporter-5f87dcb946-wt7tl /]# rbd perf image iostat rbd: waiting for initial image stats NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT pool/csi-vol-5c2d22e3-9195-11ed-aab9-a6732a88c7dd 8/s 0/s 215 KiB/s 0 B/s 14.62 ms 0.00 ns pool/csi-vol-10e31971-9196-11ed-aab9-a6732a88c7dd 5/s 0/s 79 KiB/s 0 B/s 13.18 ms 0.00 ns pool/csi-vol-0f6e0dbe-9196-11ed-aab9-a6732a88c7dd 1/s 0/s 23 KiB/s 0 B/s 20.60 ms 0.00 ns pool/csi-vol-88704e7a-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 12 KiB/s 0 B/s 12.38 ms 0.00 ns pool/csi-vol-a87c21e9-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 14 KiB/s 0 B/s 13.21 ms 0.00 ns pool/csi-vol-b6d040a2-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 8 KiB/s 0 B/s 9.88 ms 0.00 ns pool/csi-vol-efb7c236-6fc9-11ed-aab9-a6732a88c7dd 0/s 0/s 3.2 KiB/s 0 B/s 2.96 ms 0.00 ns NAME WR RD WR_BYTES RD_BYTES WR_LAT RD_LAT pool/csi-vol-5c2d22e3-9195-11ed-aab9-a6732a88c7dd 12/s 0/s 165 KiB/s 0 B/s 26.07 ms 0.00 ns pool/csi-vol-10e31971-9196-11ed-aab9-a6732a88c7dd 4/s 0/s 91 KiB/s 0 B/s 9.83 ms 0.00 ns pool/csi-vol-b6d040a2-919c-11ed-aab9-a6732a88c7dd 1/s 0/s 22 KiB/s 0 B/s 15.25 ms 0.00 ns pool/csi-vol-efb7c236-6fc9-11ed-aab9-a6732a88c7dd 1/s 0/s 38 KiB/s 0 B/s 20.40 ms 0.00 ns pool/csi-vol-88704e7a-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 9.6 KiB/s 0 B/s 12.96 ms 0.00 ns pool/csi-vol-a87c21e9-919c-11ed-aab9-a6732a88c7dd 0/s 0/s 10 KiB/s 0 B/s 3.99 ms 0.00 ns pool/csi-vol-0f6e0dbe-9196-11ed-aab9-a6732a88c7dd 0/s 0/s 2.4 KiB/s 0 B/s 7.33 ms 0.00 ns Best Regards, Liang Zheng

2 1

Any issues with podman 4.2 and Quincy?
by Vladimir Brik 13 Feb '23

13 Feb '23

Has anybody run into issues with Quincy and podman 4.2? 4x podman series are not mentioned in https://docs.ceph.com/en/quincy/cephadm/compatibility/ but podman 3x is no longer available in Alma Linux Vlad

2 1

Health warning - POOL_TARGET_SIZE_BYTES_OVERCOMMITED
by Sake Paulusma 13 Feb '23

13 Feb '23

Hello, I configured a stretched cluster on two datacenters. It's working fine, except this weekend the Raw Capicity exceeded 50% and the error POOL_TARGET_SIZE_BYTES_OVERCOMMITED showed up. The command "ceph df" is showing the correct cluster size, but "ceph osd pool autoscale-status" is showing half of the total Raw Capacity. What could be wrong? ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph status cluster: id: adbe7bb6-5h6d-11ed-8511-004449ede0c health: HEALTH_WARN 1 MDSs report oversized cache 1 subtrees have overcommitted pool target_size_bytes services: mon: 5 daemons, quorum host1,host2,host3,host4,host5 (age 4w) mgr: aqsoel11445.nqamuz(active, since 5w), standbys: host1.wujgas mds: 2/2 daemons up, 2 standby osd: 12 osds: 12 up (since 5w), 12 in (since 9w) data: volumes: 2/2 healthy pools: 5 pools, 193 pgs objects: 17.31M objects, 1.2 TiB usage: 5.0 TiB used, 3.8 TiB / 8.8 TiB avail pgs: 192 active+clean 1 active+clean+scrubbing ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 8.8 TiB 3.8 TiB 5.0 TiB 5.0 TiB 56.83 TOTAL 8.8 TiB 3.8 TiB 5.0 TiB 5.0 TiB 56.83 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 449 KiB 2 1.8 MiB 0 320 GiB cephfs.application-tst.meta 2 16 540 MiB 18.79k 2.1 GiB 0.16 320 GiB cephfs.application-tst.data 3 32 4.4 GiB 8.01k 17 GiB 1.33 320 GiB cephfs.application-acc.meta 4 16 11 GiB 3.54M 45 GiB 3.37 320 GiB cephfs.application-acc.data 5 128 1.2 TiB 13.74M 4.8 TiB 79.46 320 GiB ---------------------------------------------------------------------------- ---------------------------------------------------------------------------- [ceph: root@aqsoel11445 /]# ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 448.5k 4.0 4499G 0.0000 1.0 1 on False cephfs.application-tst.meta 539.8M 4.0 4499G 0.0005 4.0 16 on False cephfs.application-tst.data 4488M 51200M 4.0 4499G 0.0444 1.0 32 on False cephfs.application-acc.meta 11430M 4.0 4499G 0.0099 4.0 16 on False cephfs.application-acc.data 1244G 4.0 4499G 1.1062 1.0000 0.9556 1.0 128 on False ----------------------------------------------------------------------------

2 3

mds damage cannot repair
by Andrej Filipcic 13 Feb '23

13 Feb '23

Hi, there is mds damage on our cluster, version 17.2.5, [ { "damage_type": "backtrace", "id": 2287166658, "ino": 3298564401782, "path": "/hpc/home/euliz/.Xauthority" } ] The recursive repair does not fix it, ...ceph tell mds.0 scrub start /hpc/home/euliz force,repair,recursive mds log: 2023-02-10T07:01:34.012+0100 7f46df3ea700 0 mds.0.cache failed to open ino 0x30001c26a76 err -116/0 2023-02-10T07:01:34.012+0100 7f46df3ea700 0 mds.0.cache open_remote_dentry_finish bad remote dentry [dentry #0x1/hpc/home/euliz/.Xauthority [568,head] auth REMOTE(reg) (dversion lock) pv=0 v=4425667830 ino=(nil) state=1073741824 | ptrwaiter=1 0x5560eb33a780] Any clue how to fix this? or remove the file from namespace? it is not important... Thanks, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail:Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

2 2

Migrate a bucket from replicated pool to ec pool
by Boris Behrens 13 Feb '23

13 Feb '23

Hi, we use rgw as our backup storage, and it basically holds only compressed rbd snapshots. I would love to move these out of the replicated into a ec pool. I've read that I can set a default placement target for a user ( https://docs.ceph.com/en/octopus/radosgw/placement/) What does happen to the existing user data? How do I move the existing data to the new pool? Does it somehow interfere with ongoing data upload (it is one internal user, with 800 buckets which constantly get new data and old data removed)? Cheers Boris ps: Can't wait to see some of you at the cephalocon :) -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 4

[RGW - octopus] too many omapkeys on versioned bucket
by Boris Behrens 13 Feb '23

13 Feb '23

Hi, I have one bucket that showed up with a large omap warning, but the amount of objects in the bucket, does not align with the amount of omap keys. The bucket is sharded to get rid of the "large omapkeys" warning. I've counted all the omapkeys of one bucket and it came up with 33.383.622 (rados -p INDEXPOOL listomapkeys INDEXOBJECT | wc -l) I've checked the amount of actual rados objects and it came up with 17.095.877 (rados -p DATAPOOL ls | grep BUCKETMARKER | wc -l) I've checked the bucket index and it came up with 16.738.482 (radosgw-admin bi list --bucket BUCKET | grep -F '"idx":' | wc -l) I have tried to fix it with radosgw-admin bucket check --check-objects --fix --bucket BUCKET but this did not change anything. Is this a known bug or might there be something else going on. How can I investigate further? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 3

Is ceph bootstrap keyrings in use after bootstrap?
by Zhongzhou Cai 13 Feb '23

13 Feb '23

Hi, I'm on Ceph version 16.2.10, and I found there are a bunch of bootstrap keyrings (i.e., client.bootstrap-<mds|mgr|osd|rbd|rbd-mirror|rgw>) located at /var/lib/ceph/bootstrap-<mds|mgr|osd|rbd|rbd-mirror|rgw>/ceph.keyring after bootstrap. Are they still in use after bootstrap? Is it safe to remove them from host and even from ceph monitor? Thanks, Zhongzhou Cai

2 1

Health warning - POOL_TARGET_SIZE_BYTES_OVERCOMMITED
by Sake Paulusma 13 Feb '23

13 Feb '23

The RATIO for cephfs.application-acc.data shouldn't be over 1.0, I believe this triggered the error. All weekend I was thinking about this issue, but couldn't find an option to correct this. But minutes after posting I found a blog about the autoscaler (https://ceph.io/en/news/blog/2022/autoscaler_tuning) and it speaks about the option to set the Rate. Shouldn't this option be set to 2 when using a stretched cluster and not 4?

1 0

Limited set of permissions for an RGW user (S3)
by Gauvain Pocentek 13 Feb '23

13 Feb '23

Hi list, A little bit of background: we provide S3 buckets using RGW (running quincy), but users are not allowed to manage their buckets, just read and write objects in them. Buckets are created by an admin user, and read/write permissions are given to end users using S3 bucket policies. We set the users quota to 0 for everything to forbid them to create buckets. This is not really scalable and a bit annoying for the users. So we are trying to find a solution to allow users to create their own buckets but with a limited set of APIs available (no policy change for example). The ceph doc says that policies cannot be applied on users, groups or roles yet. Is there any other way to achieve this? Any feedback will be appreciated. Thanks! Gauvain

1 0

Re%3A%20%5Bceph-users%5D%20Re%3A%20Exit%20yolo%20mode%20by%20increasing%20size/min_size%20does%20not%20%28really%29%20work
by Stefan Pinter 13 Feb '23

13 Feb '23

Hi! Thank you 😊 Your message was very helpful! The main reason why “ceph df“ went to “100% USAGE” was because of the crush rule said this: "min_size": 2 "max_size": 2 And the new “size” was 3, so the rule did not want to work with the pool. After creating a new rule and setting the pools to this new rule, it seems to work (in our test cluster) rule newrule { ruleset 4 type replicated min_size 3 max_size 3 step take default step choose firstn 2 type room step chooseleaf firstn 2 type host step emit } Thanks again! Further testing now 😊 ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 0

recovery for node disaster
by farhad kh 13 Feb '23

13 Feb '23

I have a cluster of three nodes, with three replicas per pool on cluster nodes --------- HOST ADDR LABELS STATUS apcepfpspsp0101 192.168.114.157 _admin mon apcepfpspsp0103 192.168.114.158 mon _admin apcepfpspsp0105 192.168.114.159 mon _admin 3 hosts in cluster --------- # ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] --------- epoch 1033 fsid 9c35e594-2392-11ed-809a-005056ae050c created 2022-08-24T09:53:36.481866+0000 modified 2023-02-12T18:57:34.447536+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 51 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client luminous require_osd_release quincy stretch_mode_enabled false pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr pool 2 'k8s-rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 541 lfor 0/0/44 flags hashpspool,selfmanaged_snaps max_bytes 75161927680 stripe_width 0 application rbd pool 3 'k8s-cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 543 lfor 0/0/57 flags hashpspool max_bytes 5368709120 stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 4 'k8s-cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 542 lfor 0/0/57 flags hashpspool max_bytes 32212254720 stripe_width 0 application cephfs ----------- Is it possible to recover data when two nodes with all physical disks are lost for any reason? What is the maximum number of fault tolerance for the cluster? For this purpose, consider the default settings . what changes should I make to increase fault tolerance?

2 1

Subject: OSDs added, remapped pgs and objects misplaced cycling up and down
by Chris Dunlop 13 Feb '23

13 Feb '23

Hi, ceph-16.2.9 I've added some new osds - some added to existing hosts and some on newly-commissioned hosts. The new osds were added to the data side of an existing EC 8.3 pool. I've been waiting for the system to finish remapping / backfilling for some time. Originally the number of remapped pgs and objects misplaced was steadily decreasing. However in the last few days both remapped pgs and objects misplaced have increased several times. Both then steadily decrease before jumping up again. See below for some details on this. I'm guessing these are associated stats, i.e. the number of objects misplaced reflects the objects in the remapped pgs so it's not surprising they're moving in concert. I thought at first that this cycling might be due to the balancer (which was enabled), but the cycling is continuing after the balancer was disabled, e.g.: $ ts; ceph balancer status 2023-02-09 12:39:14 { "active": true, "last_optimize_duration": "0:00:00.000313", "last_optimize_started": "Thu Feb 9 01:38:55 2023", "mode": "upmap", "optimize_result": "Too many objects (0.078537 > 0.050000) are misplaced; try again later", "plans": [] } $ ts; ceph balancer off 2023-02-09 13:03:27 ceph balancer off 4 days later, the balancer is still inactive, and "last_optimize_started" indicates it hasn't run since before it was turned off: $ ts; ceph balancer status 2023-02-13 10:18:18 { "active": false, "last_optimize_duration": "0:00:00.000420", "last_optimize_started": "Thu Feb 9 02:02:56 2023", "mode": "upmap", "optimize_result": "Too many objects (0.078101 > 0.050000) are misplaced; try again later", "plans": [] } Since the balancer was turned off the cycling continues: "misplaced" gets down to 5% and then rapidly increases to 9%, in conjunction with "starting backfill" messages, before starting to slowly decrease once again. See below for a log extract. Is this "sawtooth" pattern of remapped pgs and misplaced objects a normal consequence of adding OSDs? Will it eventually settle done itself or do I need to "do something"? Is there some way of telling how much work remains until it all settles down? Cheers, Chris ------------------------------------------------------------------------------ # # General health # scrubs disabled until added OSDs complete backfilling etc. # $ ceph -s cluster: id: c6618970-0ce0-4cb2-bc9a-dd5f29b62e24 health: HEALTH_WARN noout,noscrub,nodeep-scrub flag(s) set 5945 pgs not deep-scrubbed in time 5945 pgs not scrubbed in time 5 slow ops, oldest one blocked for 464 sec, mon.k2 has slow ops services: mon: 5 daemons, quorum k2,b2,b4,k1,b5 (age 5d) mgr: b4(active, since 3M), standbys: b5, b2 osd: 82 osds: 82 up (since 2w), 82 in (since 2w); 12 remapped pgs flags noout,noscrub,nodeep-scrub data: pools: 16 pools, 5945 pgs objects: 70.35M objects, 88 TiB usage: 186 TiB used, 242 TiB / 428 TiB avail pgs: 50050192/700705500 objects misplaced (7.143%) 5933 active+clean 10 active+remapped+backfill_wait 2 active+remapped+backfilling io: client: 2.2 MiB/s rd, 1.8 MiB/s wr, 395 op/s rd, 122 op/s wr recovery: 24 MiB/s, 24 objects/s # # remapped pgs steadily decreasing then started periodically increasing # (extract from periodic "ceph -s") # 2023-02-06 00:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 26 remapped pgs 2023-02-06 04:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 25 remapped pgs 2023-02-06 06:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 24 remapped pgs 2023-02-06 09:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 6d); 23 remapped pgs 2023-02-06 12:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 22 remapped pgs 2023-02-06 14:00:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 21 remapped pgs 2023-02-06 14:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 20 remapped pgs 2023-02-06 20:30:00 osd: 82 osds: 82 up (since 7d), 82 in (since 7d); 19 remapped pgs 2023-02-07 00:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 18 remapped pgs 2023-02-07 01:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 7d); 17 remapped pgs 2023-02-07 11:00:00 osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 15 remapped pgs 2023-02-07 16:30:00 osd: 82 osds: 82 up (since 8d), 82 in (since 8d); 13 remapped pgs 2023-02-07 22:30:00 osd: 82 osds: 82 up (since 9d), 82 in (since 8d); 11 remapped pgs 2023-02-08 15:30:00 osd: 82 osds: 82 up (since 9d), 82 in (since 9d); 9 remapped pgs 2023-02-08 21:30:00 osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 15 remapped pgs <<< increase 2023-02-09 07:30:00 osd: 82 osds: 82 up (since 10d), 82 in (since 9d); 13 remapped pgs 2023-02-09 13:03:27 ceph balancer off 2023-02-09 22:30:00 osd: 82 osds: 82 up (since 11d), 82 in (since 10d); 11 remapped pgs 2023-02-10 14:00:00 osd: 82 osds: 82 up (since 11d), 82 in (since 11d); 9 remapped pgs 2023-02-10 19:00:00 osd: 82 osds: 82 up (since 11d), 82 in (since 11d); 15 remapped pgs <<< increase 2023-02-11 04:30:00 osd: 82 osds: 82 up (since 12d), 82 in (since 12d); 13 remapped pgs 2023-02-11 19:00:00 osd: 82 osds: 82 up (since 12d), 82 in (since 12d); 11 remapped pgs 2023-02-12 12:00:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 9 remapped pgs 2023-02-12 18:00:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 15 remapped pgs <<< increase 2023-02-12 20:30:00 osd: 82 osds: 82 up (since 13d), 82 in (since 13d); 14 remapped pgs 2023-02-13 04:00:00 osd: 82 osds: 82 up (since 2w), 82 in (since 13d); 12 remapped pgs # # objects misplaced increases when the remapped pgs increases # 2023-02-06 00:00:00 pgs: 62898818/685798373 objects misplaced (9.172%) 2023-02-06 04:00:00 pgs: 61995946/686058837 objects misplaced (9.037%) 2023-02-06 06:00:00 pgs: 61090940/686122303 objects misplaced (8.904%) 2023-02-06 09:00:00 pgs: 59285766/686306445 objects misplaced (8.638%) 2023-02-06 12:30:00 pgs: 57626520/686387322 objects misplaced (8.396%) 2023-02-06 14:00:00 pgs: 56813753/686388870 objects misplaced (8.277%) 2023-02-06 14:30:00 pgs: 56578343/686389185 objects misplaced (8.243%) 2023-02-06 20:30:00 pgs: 54255700/686969448 objects misplaced (7.898%) 2023-02-07 00:00:00 pgs: 53089526/687266865 objects misplaced (7.725%) 2023-02-07 01:00:00 pgs: 52810518/687391707 objects misplaced (7.683%) 2023-02-07 11:00:00 pgs: 50169973/688319189 objects misplaced (7.289%) 2023-02-07 16:30:00 pgs: 49312642/689410730 objects misplaced (7.153%) 2023-02-07 22:30:00 pgs: 48244798/689771838 objects misplaced (6.994%) 2023-02-08 15:30:00 pgs: 38058106/691166947 objects misplaced (5.506%) 2023-02-08 21:30:00 pgs: 62851681/691618387 objects misplaced (9.088%) <<< increase 2023-02-09 07:30:00 pgs: 57693559/693214621 objects misplaced (8.323%) 2023-02-09 13:03:27 ceph balancer off 2023-02-09 22:30:00 pgs: 48248474/694618590 objects misplaced (6.946%) 2023-02-10 14:00:00 pgs: 37872501/696136659 objects misplaced (5.440%) 2023-02-10 19:00:00 pgs: 64062339/696645992 objects misplaced (9.196%) <<< increase 2023-02-11 04:30:00 pgs: 52183052/699438360 objects misplaced (7.461%) 2023-02-11 19:00:00 pgs: 49167191/699631520 objects misplaced (7.028%) 2023-02-12 12:00:00 pgs: 39485735/700248360 objects misplaced (5.639%) 2023-02-12 18:00:00 pgs: 63540276/700256808 objects misplaced (9.074%) <<< increase 2023-02-12 20:30:00 pgs: 61766426/700291865 objects misplaced (8.820%) 2023-02-13 04:00:00 pgs: 57622457/700365805 objects misplaced (8.227%) --- annotated log extract --- ## ## "misplaced" decreasing... ## Feb 12 17:50:20 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:18.782191+0000 mgr.b4 (mgr.17297882) 4129637 : cluster [DBG] pgmap v4094135: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 178 op/s; 35014018/700256808 objects misplaced (5.000%); 34 MiB/s, 34 objects/s recovering Feb 12 17:50:20 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:18.782191+0000 mgr.b4 (mgr.17297882) 4129637 : cluster [DBG] pgmap v4094135: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 178 op/s; 35014018/700256808 objects misplaced (5.000%); 34 MiB/s, 34 objects/s recovering Feb 12 17:50:27 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:26.833542+0000 mgr.b4 (mgr.17297882) 4129641 : cluster [DBG] pgmap v4094141: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 2.7 MiB/s rd, 4.9 MiB/s wr, 226 op/s; 35012325/700256808 objects misplaced (5.000%); 37 MiB/s, 36 objects/s recovering Feb 12 17:50:30 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:28.848020+0000 mgr.b4 (mgr.17297882) 4129642 : cluster [DBG] pgmap v4094143: 5945 pgs: 3 active+remapped+backfilling, 6 active+remapped+backfill_wait, 5936 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 2.0 MiB/s rd, 4.9 MiB/s wr, 436 op/s; 35011830/700256808 objects misplaced (5.000%); 36 MiB/s, 35 objects/s recovering ## ## shortly after, "misplaced" has increased, followed by the "starting backfill" lines ## I'm not sure if the line ordering here is a quirk of journalctl, maybe the "starting backfill" came first which would make some sense ## Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:30.860404+0000 mgr.b4 (mgr.17297882) 4129643 : cluster [DBG] pgmap v4094145: 5945 pgs: 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5934 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 419 KiB/s rd, 1.2 MiB/s wr, 388 op/s; 44579289/700256808 objects misplaced (6.366%); 31 MiB/s, 30 objects/s recovering Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.031822+0000 osd.64 (osd.64) 362 : cluster [DBG] 33.2cs0 starting backfill to osd.25(4) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.031951+0000 osd.64 (osd.64) 363 : cluster [DBG] 33.6cs0 starting backfill to osd.25(4) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.044285+0000 osd.64 (osd.64) 364 : cluster [DBG] 33.6cs0 starting backfill to osd.51(5) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.044655+0000 osd.64 (osd.64) 365 : cluster [DBG] 33.2cs0 starting backfill to osd.51(5) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.057888+0000 osd.64 (osd.64) 366 : cluster [DBG] 33.6cs0 starting backfill to osd.63(7) from (0'0,0'0] MAX to 639282'24593871 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.058039+0000 osd.64 (osd.64) 367 : cluster [DBG] 33.2cs0 starting backfill to osd.63(7) from (0'0,0'0] MAX to 639282'24881196 Feb 12 17:50:32 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:31.077517+0000 osd.64 (osd.64) 368 : cluster [DBG] 33.6cs0 starting backfill to osd.68(2) from (0'0,0'0] MAX to 639282'24593871 ## (more "starting backfill" lines elided here) ## ## "misplaced" continues to increase rapidly to 9%... ## Feb 12 17:50:34 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:32.871242+0000 mgr.b4 (mgr.17297882) 4129644 : cluster [DBG] pgmap v4094148: 5945 pgs: 2 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5932 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 424 KiB/s rd, 1.2 MiB/s wr, 440 op/s; 54133989/700256808 objects misplaced (7.731%); 31 MiB/s, 31 objects/s recovering Feb 12 17:50:36 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:34.886503+0000 mgr.b4 (mgr.17297882) 4129645 : cluster [DBG] pgmap v4094149: 5945 pgs: 2 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5932 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 442 KiB/s rd, 1.1 MiB/s wr, 483 op/s; 54133670/700256808 objects misplaced (7.731%); 41 MiB/s, 41 objects/s recovering Feb 12 17:50:37 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:36.901531+0000 mgr.b4 (mgr.17297882) 4129646 : cluster [DBG] pgmap v4094150: 5945 pgs: 4 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 281 KiB/s rd, 902 KiB/s wr, 324 op/s; 63666090/700256808 objects misplaced (9.092%); 17 MiB/s, 17 objects/s recovering ## ## then starts to fall again... ## Feb 12 17:50:42 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:40.930869+0000 mgr.b4 (mgr.17297882) 4129648 : cluster [DBG] pgmap v4094152: 5945 pgs: 4 activating+remapped, 3 active+remapped+backfilling, 8 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 407 KiB/s rd, 1.4 MiB/s wr, 479 op/s; 63665652/700256808 objects misplaced (9.092%); 28 MiB/s, 28 objects/s recovering Feb 12 17:50:43 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:42.948421+0000 mgr.b4 (mgr.17297882) 4129649 : cluster [DBG] pgmap v4094153: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 455 KiB/s rd, 1.5 MiB/s wr, 542 op/s; 63665652/700256808 objects misplaced (9.092%); 27 MiB/s, 26 objects/s recovering Feb 12 17:50:46 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:44.961474+0000 mgr.b4 (mgr.17297882) 4129650 : cluster [DBG] pgmap v4094154: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 436 KiB/s rd, 1.4 MiB/s wr, 518 op/s; 63664579/700256808 objects misplaced (9.092%); 37 MiB/s, 37 objects/s recovering Feb 12 17:50:50 b2 ceph-c6618970-0ce0-4cb2-bc9a-dd5f29b62e24-mon-b2[1613051]: cluster 2023-02-12T06:50:48.991308+0000 mgr.b4 (mgr.17297882) 4129652 : cluster [DBG] pgmap v4094156: 5945 pgs: 3 active+remapped+backfilling, 12 active+remapped+backfill_wait, 5930 active+clean; 88 TiB data, 185 TiB used, 243 TiB / 428 TiB avail; 513 KiB/s rd, 1.6 MiB/s wr, 534 op/s; 63664502/700256808 objects misplaced (9.092%); 33 MiB/s, 33 objects/s recovering ------------------------------------------------------------------------------

2 2

Quincy: Stuck on image permissions
by hicks＠cgi.cz 12 Feb '23

12 Feb '23

Hello guys, could someone help me with this? We've been long-time CEPH users... runing several Mimic + Pacific CEPH clusters. Dozens of disk per cluster, typically. BUT... now I have this brand new Quincy cluster and I'm not able to give CLIENT (Quincy on Rocky 8) rw access to ONE IMAGE on Quincy cluster (cephadm / Rocky 9). I'm using something what worked for us for ages: rbd auth ls: client.xxx key: ... caps: [mon] profile rbd caps: [osd] allow rwx pool prod object_prefix rbd_data.600d1c6723ae; allow rwx pool prod object_prefix rbd_header.600d1c6723ae; allow rx pool prod object_prefix rbd_id.xxx-data rbd info: rbd image 'xxx-data': size 2 TiB in 524288 objects order 22 (4 MiB objects) snapshot_count: 2 id: 600d1c6723ae block_name_prefix: rbd_data.600d1c6723ae format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: rados ls: rbd_data.600d1c6723ae.000000000003958d rbd_header.600d1c6723ae rbd_id.xxx-data BUT... it DOES NOT WORK. When I try it to map on client it says: 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::GetMetadataRequest: 0x7f3a1c001f40 handle_metadata_list: failed to retrieve image metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::RefreshRequest: failed to retrieve pool metadata: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::image::OpenRequest: failed to refresh image: (1) Operation not permitted 2023-02-11T20:49:18.665+0100 7f3a337fe700 -1 librbd::ImageState: 0x555eff78cfc0 failed to open image: (1) Operation not permitted rbd: error opening image xxx-data: (1) Operation not permitted The mapping and access DOES work when I put "osd allow *" into ceph auth. What is the recommended syntax for Quincy? btw: this use case should be mentioned in the manual I think... Thanks!

2 1

Ceph Quincy On Rocky 8.x - Upgrade To Rocky 9.1
by duluxoz 11 Feb '23

11 Feb '23

Hi All, Sorry if this was mentioned previously (I obviously missed it if it was) but can we upgrade a Ceph Quincy Host/Cluster from Rocky Linux (RHEL) v8.6/8.7 to v9.1 (yet), and if so, what is / where can I find the procedure to do this - ie is there anything "special" that needs to be done because of Ceph, or can we just do a "simple" v8.x +> v9.1 upgrade? Thanks in advance Cheers Dulux-Oz

3 6

Re: Exit yolo mode by increasing size/min_size does not (really) work
by stefan 11 Feb '23

11 Feb '23

ah, thank you so much Eugen! this makes sense! i will report what we will have changed and if it worked or not :) i wish you a nice weekend!

1 0

Exit yolo mode by increasing size/min_size does not (really) work
by Stefan Pinter 10 Feb '23

10 Feb '23

Hi! 😊 It would be very kind of you to help us with that! We have pools in our ceph cluster that are set to replicated size 2 min_size 1. Obviously we want to go to size 3 / min_size 2 but we experience problems with that. USED goes to 100% instantly and MAX AVAIL goes to 0. Write operations seemed to stop. POOLS: NAME ID USED %USED MAX AVAIL OBJECTS Pool1 24 35791G 35.04 66339G 8927762 Pool2 25 11610G 14.89 66339G 3004740 Pool3 26 17557G 100.00 0 2666972 Before the change it was like this: NAME ID USED %USED MAX AVAIL OBJECTS Pool1 24 35791G 35.04 66339G 8927762 Pool2 25 11610G 14.89 66339G 3004740 Pool3 26 17558G 20.93 66339G 2667013 This was quite surprising to us as we’d expect USED to go to something like 30%. Going back to 2/1 also gave us back the 20.93% usage instantly. What’s the matter here? Thank you and best regards Stefan ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

3 5

issue in connecting Openstack(Kolla-ansible) manila with external ceph (cephadm)
by Haitham Abdulaziz 10 Feb '23

10 Feb '23

i deployed kolla-ansible & cephadm on virtual machines (kvm) . My ceph cluster is on 3 vms with 12 vCPU each and 24gb of ram i used cephadm to deploy ceph ceph -s : -------------------------- cluster: id: a0e5ad36-a54c-11ed-9aea-5254008c2a3e health: HEALTH_OK services: mon: 3 daemons, quorum ceph0,ceph1,ceph2 (age 6h) mgr: ceph0.dzutak(active, since 24h), standbys: ceph1.aizuyc mds: 3/3 daemons up, 6 standby osd: 9 osds: 9 up (since 24h), 9 in (since 24h) data: volumes: 3/3 healthy pools: 9 pools, 257 pgs objects: 70 objects, 7.3 KiB usage: 76 MiB used, 780 GiB / 780 GiB avail pgs: 257 active+clean -------------------------- my openstack deployment is AIO on a single node , now i wanna link them together so i started with manila & native cephfs thinking its the easist following this doc : https://docs.openstack.org/manila/latest/admin/cephfs_driver.html#authorizi… i created the user -------------------------- client.manila key: AQC7ot9jfiDsIxAA57fb7S6bVMnr5IadsnukHQ== caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rw pool=ganesha_rados_store and created a file system called manila -------------------------- my ceph.conf -------------------------- [global] fsid = a0e5ad36-a54c-11ed-9aea-5254008c2a3e mon_host = [v2:192.168.122.25:3300/0,v1:192.168.122.25:6789/0] [v2:192.168.122.115:3300/0,v1:192.168.122.115:6789/0] [v2:192.168.122.14:3300/0,v1:192.168.122.14:6789/0] -------------------------- i moved the files as to the openstack node and trying to connect them together but it didnt go will , Viewing the logs shows -------------------------- <AIO@cephfsnative1: manila.exception.ShareBackendException: json_command failed - prefix=fs volume ls, argdict={'format': 'json'} - exception message: Bad target type 'mon-mgr'. --------------------------where should i start to fix this issue ?

2 1