January 2023 - ceph-users

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 11 Jun '24

11 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 26 Mar '24

26 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 01 Jun '23

01 Jun '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

deploying Ceph using FQDN for MON / MDS Services
by Lokendra Rathour 21 Apr '23

21 Apr '23

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

7 20

ceph/daemon stable tag
by Jonas Nemeikšis 15 Apr '23

15 Apr '23

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

2 6

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 26 Mar '23

26 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

Stuck OSD service specification - can't remove
by David Orman 15 Mar '23

15 Mar '23

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

3 4

OSD upgrade problem nautilus->octopus - snap_mapper upgrade stuck
by Jan Pekař - Imatic 28 Feb '23

28 Feb '23

Hi all, I have problem upgrading nautilus to octopus on my OSD. Upgrade mon and mgr was OK and first OSD stuck on 2023-01-12T09:25:54.122+0100 7f49ff3eae00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) and there were no activity after that for more than 48 hours. No disk activity. I restarted OSD many times and nothing changed. It is old, filestore OSD based on XFS filesystem. Is upgrade to snap mapper 2 reliable? What is OSD waiting for? Can I start OSD without upgrade and get cluster healthy with old snap structure? Or should I skip octopus upgrade and go to pacific directly (some bug backport is missing?). Thank you for help, I'm sending some logs below.. Log shows 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 ceph version 15.2.17 (694d03a6f6c6e9f814446223549caf9a9f60dba0) octopus (stable), process ceph-osd, pid 2566563 2023-01-09T19:12:49.471+0100 7f41f60f1e00 0 pidfile_write: ignore empty --pid-file 2023-01-09T19:12:49.499+0100 7f41f60f1e00 -1 missing 'type' file, inferring filestore from current/ dir 2023-01-09T19:12:49.531+0100 7f41f60f1e00 0 starting osd.0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2023-01-09T19:12:49.531+0100 7f41f60f1e00 -1 Falling back to public interface 2023-01-09T19:12:49.871+0100 7f41f60f1e00 0 load: jerasure load: lrc load: isa 2023-01-09T19:12:49.875+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 osd.0:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) 2023-01-09T19:12:49.883+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) backend xfs (magic 0x58465342) 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: SEEK_DATA/SEEK_HOLE is disabled via 'filestore seek data hole' config option 2023-01-09T19:12:49.927+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: splice() is disabled via 'filestore splice' config option 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2023-01-09T19:12:49.983+0100 7f41f60f1e00 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-0) detect_feature: extsize is disabled by conf 2023-01-09T19:12:50.015+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) start omap initiation 2023-01-09T19:12:50.079+0100 7f41f60f1e00 1 leveldb: Recovering log #165531 2023-01-09T19:12:50.083+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: started 2023-01-09T19:12:50.235+0100 7f41f60f1e00 1 leveldb: Level-0 table #165533: 1598 bytes OK 2023-01-09T19:12:50.583+0100 7f41f60f1e00 1 leveldb: Delete type=0 #165531 2023-01-09T19:12:50.615+0100 7f41f60f1e00 1 leveldb: Delete type=3 #165529 2023-01-09T19:12:51.339+0100 7f41f60f1e00 0 filestore(/var/lib/ceph/osd/ceph-0) mount(1861): enabling WRITEAHEAD journal mode: checkpoint is not enabled 2023-01-09T19:12:51.379+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.931+0100 7f41f60f1e00 -1 journal do_read_entry(243675136): bad header magic 2023-01-09T19:12:51.939+0100 7f41f60f1e00 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 35: 2998927360 bytes, block size 4096 bytes, directio = 1, aio = 1 2023-01-09T19:12:51.943+0100 7f41f60f1e00 1 filestore(/var/lib/ceph/osd/ceph-0) upgrade(1466) 2023-01-09T19:12:52.015+0100 7f41f60f1e00 1 osd.0 126556 init upgrade snap_mapper (first start as octopus) lsof shows COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ceph-osd 225860 ceph cwd DIR 9,127 4096 2 / ceph-osd 225860 ceph rtd DIR 9,127 4096 2 / ceph-osd 225860 ceph txt REG 9,127 31762544 5021 /usr/bin/ceph-osd ceph-osd 225860 ceph mem REG 8,70 2147237 68104224 /var/lib/ceph/osd/ceph-0/current/omap/165546.ldb ceph-osd 225860 ceph mem REG 8,70 2147792 68104190 /var/lib/ceph/osd/ceph-0/current/omap/165545.ldb ceph-osd 225860 ceph mem REG 8,70 2147689 68104240 /var/lib/ceph/osd/ceph-0/current/omap/165466.ldb ceph-osd 225860 ceph mem REG 8,70 2142721 68102679 /var/lib/ceph/osd/ceph-0/current/omap/165544.ldb ceph-osd 225860 ceph mem REG 8,70 2142677 68104239 /var/lib/ceph/osd/ceph-0/current/omap/165465.ldb ceph-osd 225860 ceph mem REG 8,70 2144979 68078254 /var/lib/ceph/osd/ceph-0/current/omap/165543.ldb ceph-osd 225860 ceph mem REG 8,70 2143705 68163491 /var/lib/ceph/osd/ceph-0/current/omap/165526.ldb ceph-osd 225860 ceph mem REG 8,70 2141468 68163492 /var/lib/ceph/osd/ceph-0/current/omap/165527.ldb ceph-osd 225860 ceph mem REG 8,70 145986 68018644 /var/lib/ceph/osd/ceph-0/current/omap/165541.ldb ceph-osd 225860 ceph mem REG 8,70 2143434 68163490 /var/lib/ceph/osd/ceph-0/current/omap/165525.ldb ceph-osd 225860 ceph mem REG 8,70 2136002 68122351 /var/lib/ceph/osd/ceph-0/current/omap/165472.ldb ceph-osd 225860 ceph mem REG 8,70 1965262 68119647 /var/lib/ceph/osd/ceph-0/current/omap/165467.ldb ceph-osd 225860 ceph mem REG 8,70 2145206 68104229 /var/lib/ceph/osd/ceph-0/current/omap/165464.ldb ceph-osd 225860 ceph mem REG 8,70 61600 68002130 /var/lib/ceph/osd/ceph-0/current/omap/165536.ldb ceph-osd 225860 ceph mem REG 8,70 352689 67945734 /var/lib/ceph/osd/ceph-0/current/omap/165530.ldb ceph-osd 225860 ceph mem REG 8,70 82500 67480519 /var/lib/ceph/osd/ceph-0/current/omap/165539.ldb ceph-osd 225860 ceph mem REG 9,127 189200 2878 /usr/lib/ceph/erasure-code/libec_isa.so ceph-osd 225860 ceph mem REG 9,127 2021136 3033 /usr/lib/ceph/erasure-code/libec_lrc.so ceph-osd 225860 ceph mem REG 9,127 360544 2942 /usr/lib/ceph/erasure-code/libec_jerasure.so ceph-osd 225860 ceph mem REG 9,127 55792 15227 /lib/x86_64-linux-gnu/libnss_files-2.28.so ceph-osd 225860 ceph mem REG 9,127 22368 133997 /usr/lib/x86_64-linux-gnu/liburcu-common.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 42976 133996 /usr/lib/x86_64-linux-gnu/liburcu-cds.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 51128 149821 /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0.0.0 ceph-osd 225860 ceph mem REG 8,70 33977 68116674 /var/lib/ceph/osd/ceph-0/current/omap/165547.ldb ceph-osd 225860 ceph DEL REG 0,19 64261192 /[aio] ceph-osd 225860 ceph DEL REG 0,19 64261191 /[aio] ceph-osd 225860 ceph mem REG 9,127 158400 2523 /lib/x86_64-linux-gnu/liblzma.so.5.2.4 ceph-osd 225860 ceph mem REG 9,127 135064 12635 /lib/x86_64-linux-gnu/libnl-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 488272 133248 /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200.26.0 ceph-osd 225860 ceph mem REG 9,127 35808 15263 /lib/x86_64-linux-gnu/librt-2.28.so ceph-osd 225860 ceph mem REG 9,127 55512 133042 /usr/lib/x86_64-linux-gnu/libunwind.so.8.0.1 ceph-osd 225860 ceph mem REG 9,127 30776 16463 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 ceph-osd 225860 ceph mem REG 9,127 1820400 15070 /lib/x86_64-linux-gnu/libc-2.28.so ceph-osd 225860 ceph mem REG 9,127 100712 7687 /lib/x86_64-linux-gnu/libgcc_s.so.1 ceph-osd 225860 ceph mem REG 9,127 1579448 15202 /lib/x86_64-linux-gnu/libm-2.28.so ceph-osd 225860 ceph mem REG 9,127 1570256 133576 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 ceph-osd 225860 ceph mem REG 9,127 112800 133289 /usr/lib/x86_64-linux-gnu/librdmacm.so.1.1.22.1 ceph-osd 225860 ceph mem REG 9,127 104496 133287 /usr/lib/x86_64-linux-gnu/libibverbs.so.1.5.22.1 ceph-osd 225860 ceph mem REG 9,127 149704 5527 /lib/x86_64-linux-gnu/libudev.so.1.6.13 ceph-osd 225860 ceph mem REG 9,127 146968 15251 /lib/x86_64-linux-gnu/libpthread-2.28.so ceph-osd 225860 ceph mem REG 9,127 3044224 135371 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 ceph-osd 225860 ceph mem REG 9,127 14592 15128 /lib/x86_64-linux-gnu/libdl-2.28.so ceph-osd 225860 ceph mem REG 9,127 93000 15255 /lib/x86_64-linux-gnu/libresolv-2.28.so ceph-osd 225860 ceph mem REG 9,127 301288 134118 /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3 ceph-osd 225860 ceph mem REG 9,127 14184 134682 /usr/lib/x86_64-linux-gnu/libaio.so.1.0.1 ceph-osd 225860 ceph mem REG 9,127 117184 2738 /lib/x86_64-linux-gnu/libz.so.1.2.11 ceph-osd 225860 ceph mem REG 9,127 121184 136027 /usr/lib/x86_64-linux-gnu/liblz4.so.1.8.3 ceph-osd 225860 ceph mem REG 9,127 31104 133334 /usr/lib/x86_64-linux-gnu/libsnappy.so.1.1.7 ceph-osd 225860 ceph mem REG 9,127 378160 133086 /usr/lib/x86_64-linux-gnu/libleveldb.so.1d.20 ceph-osd 225860 ceph mem REG 9,127 256120 7650 /lib/x86_64-linux-gnu/libfuse.so.2.9.9 ceph-osd 225860 ceph mem REG 9,127 343008 16890 /lib/x86_64-linux-gnu/libblkid.so.1.1.0 ceph-osd 225860 ceph mem REG 8,70 1546 68022625 /var/lib/ceph/osd/ceph-0/current/omap/165542.ldb ceph-osd 225860 ceph mem REG 8,70 1598 67109142 /var/lib/ceph/osd/ceph-0/current/omap/165533.ldb ceph-osd 225860 ceph mem REG 9,127 35112 133884 /usr/lib/x86_64-linux-gnu/liburcu-bp.so.6.0.0 ceph-osd 225860 ceph mem REG 9,127 165632 15000 /lib/x86_64-linux-gnu/ld-2.28.so ceph-osd 225860 ceph 0r CHR 1,3 0t0 6 /dev/null ceph-osd 225860 ceph 1u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 2u unix 0x00000000ab7b4943 0t0 64259884 type=STREAM ceph-osd 225860 ceph 3u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 4r FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 5w FIFO 0,13 0t0 64259943 pipe ceph-osd 225860 ceph 6u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 7r FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 8w FIFO 0,13 0t0 64259944 pipe ceph-osd 225860 ceph 9u a_inode 0,14 0 11090 [eventpoll] ceph-osd 225860 ceph 10r FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 11w FIFO 0,13 0t0 64259945 pipe ceph-osd 225860 ceph 12w REG 9,127 8765 3314 /var/log/ceph/ceph-osd.0.log ceph-osd 225860 ceph 13r FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 14w FIFO 0,13 0t0 64259950 pipe ceph-osd 225860 ceph 15u unix 0x0000000097fdc46a 0t0 64259951 /var/run/ceph/ceph-osd.0.asok type=STREAM ceph-osd 225860 ceph 24r FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 25w FIFO 0,13 0t0 64261188 pipe ceph-osd 225860 ceph 26r FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 27w FIFO 0,13 0t0 64261189 pipe ceph-osd 225860 ceph 28uW REG 8,70 37 21 /var/lib/ceph/osd/ceph-0/fsid ceph-osd 225860 ceph 29r DIR 8,70 262 16 /var/lib/ceph/osd/ceph-0 ceph-osd 225860 ceph 30r DIR 8,70 24576 27 /var/lib/ceph/osd/ceph-0/current ceph-osd 225860 ceph 31u REG 8,70 10 28 /var/lib/ceph/osd/ceph-0/current/commit_op_seq ceph-osd 225860 ceph 32uW REG 8,70 0 67108882 /var/lib/ceph/osd/ceph-0/current/omap/LOCK ceph-osd 225860 ceph 33w REG 8,70 813568 68035819 /var/lib/ceph/osd/ceph-0/current/omap/165540.log ceph-osd 225860 ceph 34w REG 8,70 32068 68072533 /var/lib/ceph/osd/ceph-0/current/omap/MANIFEST-165538 ceph-osd 225860 ceph 36u BLK 8,65 0t0 422 /dev/sde1 ceph-osd 225860 ceph 37u REG 8,70 538 134217745 /var/lib/ceph/osd/ceph-0/current/meta/DIR_E/DIR_D/DIR_C/osd\\usuperblock__0_23C2FCDE__none ceph-osd 225860 ceph 38u REG 8,70 22453 1075330317 /var/lib/ceph/osd/ceph-0/current/meta/DIR_4/DIR_B/DIR_4/osdmap.126556__0_DF0524B4__none strace shows strace: Process 225860 attached futex(0x555f3e1c05c8, FUTEX_WAIT_PRIVATE, 0, NULL -- ============ Ing. Jan Pekař jan.pekar(a)imatic.cz ---- Imatic | Jagellonská 14 | Praha 3 | 130 00 https://www.imatic.cz | +420326555326 ============ --

3 8

MDS stuck in "up:replay"
by Thomas Widhalm 24 Feb '23

24 Feb '23

Hi, I'm really lost with my Ceph system. I built a small cluster for home usage which has two uses for me: I want to replace an old NAS and I want to learn about Ceph so that I have hands-on experience. We're using it in our company but I need some real-life experience without risking any company or customers data. That's my preferred way of learning. The cluster consists of 3 Raspberry Pis plus a few VMs running on Proxmox. I'm not using Proxmox' built in Ceph because I want to focus on Ceph and not just use it as a preconfigured tool. All hosts are running Fedora (x86_64 and arm64) and during an Upgrade from F36 to F37 my cluster suddenly showed all PGs as unavailable. I worked nearly a week to get it back online and I learned a lot about Ceph management and recovery. The cluster is back but I still can't access my data. Maybe you can help me? Here are my versions: [ceph: root@ceph04 /]# ceph versions { "mon": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "mgr": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 5 }, "mds": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 4 }, "overall": { "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 15 } } Here's MDS status output of one MDS: [ceph: root@ceph04 /]# ceph tell mds.mds01.ceph05.pqxmvt status 2023-01-14T15:30:28.607+0000 7fb9e17fa700 0 client.60986454 ms_handle_reset on v2:192.168.23.65:6800/2680651694 2023-01-14T15:30:28.640+0000 7fb9e17fa700 0 client.60986460 ms_handle_reset on v2:192.168.23.65:6800/2680651694 { "cluster_fsid": "ff6e50de-ed72-11ec-881c-dca6325c2cc4", "whoami": 0, "id": 60984167, "want_state": "up:replay", "state": "up:replay", "fs_name": "cephfs", "replay_status": { "journal_read_pos": 0, "journal_write_pos": 0, "journal_expire_pos": 0, "num_events": 0, "num_segments": 0 }, "rank_uptime": 1127.54018615, "mdsmap_epoch": 98056, "osdmap_epoch": 12362, "osdmap_epoch_barrier": 0, "uptime": 1127.957307273 } It's staying like that for days now. If there was a counter moving, I just would wait but it doesn't change anything and alle stats says, the MDS aren't working at all. The symptom I have is that Dashboard and all other tools I use say, it's more or less ok. (Some old messages about failed daemons and scrubbing aside). But I can't mount anything. When I try to start a VM that's on RDS I just get a timeout. And when I try to mount a CephFS, mount just hangs forever. Whatever command I give MDS or journal, it just hangs. The only thing I could do, was take all CephFS offline, kill the MDS's and do a "ceph fs reset <fs name> --yes-i-really-mean-it". After that I rebooted all nodes, just to be sure but I still have no access to data. Could you please help me? I'm kinda desperate. If you need any more information, just let me know. Cheers, Thomas -- Thomas Widhalm Lead Systems Engineer NETWAYS Professional Services GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB34510 https://www.netways.de | thomas.widhalm(a)netways.de ** stackconf 2023 - September - https://stackconf.eu ** ** OSMC 2023 - November - https://osmc.de ** ** New at NWS: Managed Database - https://nws.netways.de/managed-database ** ** NETWAYS Web Services - https://nws.netways.de **

7 23

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 24 Feb '23

24 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

Problem with IO after renaming File System .data pool
by murilo＠evocorp.com.br 23 Feb '23

23 Feb '23

Good morning everyone. On this Thursday night we went through an accident, where they accidentally renamed the .data pool of a File System making it instantly inaccessible, when renaming it again to the correct name it was possible to mount and list the files, but could not read or write. When trying to write, the FS returned as Read Only, when trying to read it returned Operation not allowed. After a period of breaking my head I tried to mount with the ADMIN user and everything worked correctly. I tried to remove the authentication of the current user through `ceph auth rm`, I created a new user through `ceph fs authorize <fs_name> client.<user> / rw` and it continued the same way, I also tried to recreate it through `ceph auth get-or-create` and nothing different happened, it stayed exactly the same. After setting `allow *` in mon, mds and osd I was able to mount, read and write again with the new user. I can understand why the File System stopped after renaming the pool, what I don't understand is why users are unable to perform operations on FS even with RW or any other user created. What could have happened behind the scenes to not be able to perform IO even with the correct permissions? Or did I apply incorrect permissions that caused this problem? Right now everything is working, I would really like to understand what happened, because I didn't find anything documented about this type of incident.

3 2

17.2.5 ceph fs status: AssertionError
by Robert Sander 23 Feb '23

23 Feb '23

Hi, I have a healthy (test) cluster running 17.2.5: root@cephtest20:~# ceph status cluster: id: ba37db20-2b13-11eb-b8a9-871ba11409f6 health: HEALTH_OK services: mon: 3 daemons, quorum cephtest31,cephtest41,cephtest21 (age 2d) mgr: cephtest22.lqzdnk(active, since 4d), standbys: cephtest32.ybltym, cephtest42.hnnfaf mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 48 osds: 48 up (since 4d), 48 in (since 4M) rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 6 portals active (3 hosts) data: volumes: 1/1 healthy pools: 17 pools, 513 pgs objects: 28.25k objects, 4.7 GiB usage: 26 GiB used, 4.7 TiB / 4.7 TiB avail pgs: 513 active+clean io: client: 4.3 KiB/s rd, 170 B/s wr, 5 op/s rd, 0 op/s wr CephFS is mounted and can be used without any issue. But I get an error when I when querying its status: root@cephtest20:~# ceph fs status Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1757, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/status/module.py", line 159, in handle_fs_status assert metadata AssertionError The dashboard's filesystem page shows no error and displays all information about cephfs. Where does this AssertionError come from? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 3

Very slow snaptrim operations blocking client I/O
by Victor Rodriguez 20 Feb '23

20 Feb '23

Hello, Asking for help with an issue. Maybe someone has a clue about what's going on. Using ceph 15.2.17 on Proxmox 7.3. A big VM had a snapshot and I removed it. A bit later, nearly half of the PGs of the pool entered snaptrim and snaptrim_wait state, as expected. The problem is that such operations ran extremely slow and client I/O was nearly nothing, so all VMs in the cluster got stuck as they could not I/O to the storage. Taking and removing big snapshots is a normal operation that we do often and this is the first time I see this issue in any of my clusters. Disks are all Samsung PM1733 and network is 25G. It gives us plenty of performance for the use case and never had an issue with the hardware. Both disk I/O and network I/O was very low. Still, client I/O seemed to get queued forever. Disabling snaptrim (ceph osd set nosnaptrim) stops any active snaptrim operation and client I/O resumes back to normal. Enabling snaptrim again makes client I/O to almost halt again. I've been playing with some settings: ceph tell 'osd.*' injectargs '--osd-max-trimming-pgs 1' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep 30' ceph tell 'osd.*' injectargs '--osd-snap-trim-sleep-ssd 30' ceph tell 'osd.*' injectargs '--osd-pg-max-concurrent-snap-trims 1' None really seemed to help. Also tried restarting OSD services. This cluster was upgraded from 14.2.x to 15.2.17 a couple of months. Is there any setting that must be changed which may cause this problem? I have scheduled a maintenance window, what should I look for to diagnose this problem? Any help is very appreciated. Thanks in advance. Victor

8 18

PSA: Potential problems in a recent kernel?
by Matthew Booth 14 Feb '23

14 Feb '23

My ceph cluster became unstable yesterday after zincati (CoreOS's auto-updater) updated one of my nodes from 37.20221225.3.0 to 37.20230110.3.1(*). The symptom was slow ops in my cephfs mds which started immediately the OSDs on this node became in and up. Excluding the OSDs on this node worked round the problem. Note that the node is also running a mon and client workloads which use ceph. Also note that the OSD came up and (IIUC) were participating in recovering their data to other OSDs. The problem only started when I allowed them to be in. I rolled back the OS update and the problem was immediately resolved. Unfortunately I didn't keep the OSD logs, but they lead me to this thread from ceph-users: https://www.mail-archive.com/ceph-users@ceph.io/msg18474.html . I wonder if we have an issue with a very recent kernel update. I should be able to reproduce if it's likely to be of use to anybody, but for now I've rolled back this OS update and disabled automatic updating on my other nodes. Matt (*) The complete list of changes: $ rpm-ostree db diff d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05 fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161 ostree diff commit from: d477f98d52bf707d4282f6835b85bed3d60e305a0cf6eb8effd4db4b89607f05 ostree diff commit to: fc214c16d248686d4cf2bb3050b59c559f091692d7af3b07ef896f1b8ab2f161 Upgraded: bash 5.2.9-3.fc37 -> 5.2.15-1.fc37 btrfs-progs 6.0.2-1.fc37 -> 6.1.2-1.fc37 clevis 18-12.fc37 -> 18-14.fc37 clevis-dracut 18-12.fc37 -> 18-14.fc37 clevis-luks 18-12.fc37 -> 18-14.fc37 clevis-systemd 18-12.fc37 -> 18-14.fc37 container-selinux 2:2.193.0-1.fc37 -> 2:2.198.0-1.fc37 containerd 1.6.12-1.fc37 -> 1.6.14-2.fc37 containers-common 4:1-73.fc37 -> 4:1-76.fc37 containers-common-extra 4:1-73.fc37 -> 4:1-76.fc37 coreutils 9.1-6.fc37 -> 9.1-7.fc37 coreutils-common 9.1-6.fc37 -> 9.1-7.fc37 crun 1.7.2-2.fc37 -> 1.7.2-3.fc37 curl 7.85.0-4.fc37 -> 7.85.0-5.fc37 dnsmasq 2.87-3.fc37 -> 2.88-1.fc37 ethtool 2:6.0-1.fc37 -> 2:6.1-1.fc37 fwupd 1.8.8-1.fc37 -> 1.8.9-1.fc37 git-core 2.38.1-1.fc37 -> 2.39.0-1.fc37 grub2-common 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-efi-x64 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-pc 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-pc-modules 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-tools 1:2.06-63.fc37 -> 1:2.06-72.fc37 grub2-tools-minimal 1:2.06-63.fc37 -> 1:2.06-72.fc37 kernel 6.0.15-300.fc37 -> 6.0.18-300.fc37 kernel-core 6.0.15-300.fc37 -> 6.0.18-300.fc37 kernel-modules 6.0.15-300.fc37 -> 6.0.18-300.fc37 libcurl-minimal 7.85.0-4.fc37 -> 7.85.0-5.fc37 libgpg-error 1.45-2.fc37 -> 1.46-1.fc37 libgusb 0.4.2-1.fc37 -> 0.4.3-1.fc37 libksba 1.6.2-1.fc37 -> 1.6.3-1.fc37 libpcap 14:1.10.1-4.fc37 -> 14:1.10.2-1.fc37 libpwquality 1.4.4-11.fc37 -> 1.4.5-1.fc37 libsmbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 libwbclient 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 moby-engine 20.10.20-1.fc37 -> 20.10.21-1.fc37 ncurses 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 ncurses-base 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 ncurses-libs 6.3-3.20220501.fc37 -> 6.3-4.20220501.fc37 net-tools 2.0-0.63.20160912git.fc37 -> 2.0-0.64.20160912git.fc37 rpm-ostree 2022.16-2.fc37 -> 2022.19-2.fc37 rpm-ostree-libs 2022.16-2.fc37 -> 2022.19-2.fc37 samba-client-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 samba-common 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 samba-common-libs 2:4.17.4-0.fc37 -> 2:4.17.4-2.fc37 selinux-policy 37.16-1.fc37 -> 37.17-1.fc37 selinux-policy-targeted 37.16-1.fc37 -> 37.17-1.fc37 tpm2-tss 3.2.0-3.fc37 -> 3.2.1-1.fc37 Removed: cracklib-dicts-2.9.7-30.fc37.x86_64 -- Matthew Booth

2 1

Corrupt bluestore after sudden reboot (17.2.5)
by Peter van Heusden 10 Feb '23

10 Feb '23

Due to the ongoing South African energy crisis <https://en.wikipedia.org/wiki/South_African_energy_crisis> our datacenter experienced sudden power loss. We are running ceph 17.2.5 deployed with cephadm. Two of our OSDs did not start correctly, with the error: # ceph-bluestore-tool fsck --path /var/lib/ceph/ed7b2c16-b053-45e2-a1fe-bf3474f90508/osd.27/ 2023-01-15T08:38:04.289+0200 7f2a2a03c540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)' thread 7f2a2a03c540 time 2023-01-15T08:39:31.304968+0200 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: 18968: FAILED ceph_assert(collection_ref) 2023-01-15T08:39:31.298+0200 7f2a2a03c540 -1 bluestore::NCB::read_allocation_from_onodes::stray object 2#55:ffffffff:::2000055f327.00002287:head# not owned by any collection ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f2a2acc07c6] 2: /usr/lib/ceph/libceph-common.so.2(+0x27c9d8) [0x7f2a2acc09d8] 3: (BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0xa24) [0x560d6baf5754] 4: (BlueStore::reconstruct_allocations(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0x5f) [0x560d6baf66ff] 5: (BlueStore::read_allocation_from_drive_on_startup()+0x99) [0x560d6baf68b9] 6: (BlueStore::_init_alloc(std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >*)+0xaca) [0x560d6bb0c15a] 7: (BlueStore::_open_db_and_around(bool, bool)+0x35c) [0x560d6bb380dc] 8: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x250) [0x560d6bb3a8c0] 9: main() 10: __libc_start_main() 11: _start() *** Caught signal (Aborted) ** in thread 7f2a2a03c540 thread_name:ceph-bluestore- 2023-01-15T08:39:31.306+0200 7f2a2a03c540 -1 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)' thread 7f2a2a03c540 time 2023-01-15T08:39:31.304968+0200 /build/ceph-17.2.5/src/os/bluestore/BlueStore.cc: 18968: FAILED ceph_assert(collection_ref) (complete log https://gist.github.com/pvanheus/5c57455cacdc91afc9ce27fd489cae25) Is there a way to recover from this? Or should I accept the OSDs as lost and rebuild them? Thanks, Peter

2 2

Permanently ignore some warning classes
by Nicola Mori 09 Feb '23

09 Feb '23

Dear Ceph users, my cluster is build with old hardware on a gigabit network, so I often experience warnings like OSD_SLOW_PING_TIME_BACK. These in turn triggers alert mails too often, forcing me to disable alerts which is not sustainable. So my question is: is it possible to tell Ceph to ignore (or at least to not send alerts for) a given class of warnings? Thank you, Nicola

2 5

Replacing OSD with containerized deployment
by mailing-lists 08 Feb '23

08 Feb '23

Dear Ceph-Users, i am struggling to replace a disk. My ceph-cluster is not replacing the old OSD even though I did: ceph orch osd rm 232 --replace The OSD 232 is still shown in the osd list, but the new hdd will be placed as a new OSD. This wouldnt mind me much, if the OSD was also placed on the bluestoreDB / NVME, but it doesn't. My steps: "ceph orch osd rm 232 --replace" remove the failed hdd. add the new one. Convert the disk within the servers bios, so that the node can have direct access on it. It shows up as /dev/sdt, enter maintenance mode reboot server drive is now /dev/sdm (which the old drive had) "ceph orch device zap node-x /dev/sdm " A new OSD is placed on the cluster. Can you give me a hint, where did I take a wrong turn? Why is the disk not being used as OSD 232? Best Ken

3 16

Re: Telemetry service is temporarily down
by Yaarit Hatuka 06 Feb '23

06 Feb '23

Hi everyone, Our telemetry service is up and running again. Thanks Adam Kraitman and Dan Mick for restoring the service. We thank you for your patience and appreciate your contribution to the project! Thanks, Yaarit On Tue, Jan 3, 2023 at 3:14 PM Yaarit Hatuka <yhatuka(a)redhat.com> wrote: > Hi everyone, > > We are having some infrastructure issues with our telemetry backend, and > we are working on fixing it. > Thanks Jan Horacek for opening this issue > <https://tracker.ceph.com/issues/58371> [1]. We will update once the > service is back up. > We are sorry for any inconvenience you may be experiencing, and appreciate > your patience. > > Thanks, > Yaarit > > [1] https://tracker.ceph.com/issues/58371 >

1 2

January Ceph Science Virtual User Group
by Kevin Hrpcek 02 Feb '23

02 Feb '23

Hey all, We will be having a Ceph science/research/big cluster call on Tuesday January 31st. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20230131 Ceph calendar event details: January 31, 2023 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc <http://bjn.vc> 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

4 3

How to get RBD client log?
by Jinhao Hu 02 Feb '23

02 Feb '23

Hi, How can I collect the logs of the RBD client?

3 3

Re: Debian update to 16.2.11-1~bpo11+1 failing
by Matthias Aebi 01 Feb '23

01 Feb '23

I can only confirm that. The file https://download.ceph.com/debian-pacific/pool/main/c/ceph/python3-rados_16.… is clearly missing on the ceph download server which makes it impossible to install the upgrade on Debian. And as the previous 16.2.10 Package definition has been replaced with the latest one, this currently makes it impossible to install Pacific on a Debian system.

3 2

Write amplification for CephFS?
by Manuel Holtgrewe 30 Jan '23

30 Jan '23

Dear all, is there information available anywhere about the write amplification for CephFS? I found quite some material on write amplification of VMs using journaled file system on top of RBD but nothing as it relates to CephFS? From my understanding I would expect the following: - for X-rep, data needs to be written to X pgs (factor X) - for k-m EC, data is written to k+m pgs (factor (k+m)/k) Am I correct? How much is added by meta data, WAL? Do I miss anything? Best wishes, Manuel

2 5

rbd online sparsify image
by Jiatong Shen 30 Jan '23

30 Jan '23

Hello community experts, I would like to know the status of rbd image sparsify. From the website, it should be added at Nautilus ( https://docs.ceph.com/en/latest/releases/nautilus/ from pr (26226 <https://github.com/ceph/ceph/pull/26226>,) ) but on mimic it is related again https://docs.ceph.com/en/latest/releases/octopus/) Is it still the same commit? or is there any update/bug fixes that is not backported to 14.x? I am also curious why it is named re-sparsify in the pr title? Thank you very much for the help. -- Best Regards, Jiatong Shen

2 3

OSDs will not start
by Geoffrey Rhodes 30 Jan '23

30 Jan '23

Good day all, I've an issue with a few OSDs (in two different nodes) that attempt to start but fail / crash quite quickly. They are all LVM disks. I've tried upgrading software, health checks on the hardware (nodes and disks) and there doesn't seem to be any issues there. Recently I've had a few "other" disks physically fail in the cluster and now have one PG down which is blocking some IO on CephFS. I've added the output of the osd journalctl and the osd log below in case it's helpful to identify anything obvious. I also set debug bluefs = 20 , saw this in another post. I recently manually upgraded this node to (17.2.0) before the problem began, later to (17.2.5). - The other osds in this node start / run fine. The other node (15.2.17) also has a few osds that will not start and some that run without issue. Could anyone point me in the right direction to investigate and solve my osd issues. https://pastebin.com/3PkCabdf https://pastebin.com/BT9bnhSb Production system mainly used for CephFS OS: Ubuntu 20.04.5 LTS Ceph versions: 15.2.17 - Octopus (one OSD node manually upgraded to 17.2.5 - Quincy) Erasure data pool (K=4, M=2) - The journal's for each osd are co-located on each drive Kind regards Geoffrey Rhodes

1 1

rbd-mirror ceph quincy Not able to find rbd_mirror_journal_max_fetch_bytes config in rbd mirror
by ankit raikwar 30 Jan '23

30 Jan '23

Hello All, In the ceph quincy Not able to find rbd_mirror_journal_max_fetch_bytes config in rbd mirror i configured the ceph cluster almost 400 tb and enable the rbd-mirror in the starting stage i'm able to achive the almost 9 GB speed , but after the rebalane completed of the all the images . rbd-mirror speed got automaticily reduce to between 4 to 5 mbps. in my primary cluster we are continuelsy writing the 50 to 400 mbps data but replication speed only we get the 4 to 5 mbps. also we have the 10 Gbps replication network bandwidth. Note::- I also try to find the option rbd_mirror_journal_max_fetch_bytes but i'm not able to find the this option in the configuration. also when i try to set from the command line it's showing error like command: ceph config set client.rbd rbd_mirror_journal_max_fetch_bytes 33554432 error: Error EINVAL: unrecognized config option 'rbd_mirror_journal_max_fetch_bytes' cluster version ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) Please suggest any alternative way to configurre this option or how i can improve the replication n/w speed.

2 5

Real memory usage of the osd(s)
by Szabo, Istvan (Agoda) 30 Jan '23

30 Jan '23

Hello, If buffered_io is enabled, is there a way to know what is the exactly used physical memory from each osd? What I've found is the dump_mempools which last entries are the following, but this bytes would be the real physical memory usage? "total": { "items": 60005205, "bytes": 995781359 Also which metric is this value? I haven't found any. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

All pgs unknown
by Daniel Brunner 29 Jan '23

29 Jan '23

Hi, my ceph cluster started to show HEALTH_WARN, there are no healthy pgs left, all are unknown, but it seems my cephfs is still readable, how to investigate this any further? $ sudo ceph -s cluster: id: ddb7ebd8-65b5-11ed-84d7-22aca0408523 health: HEALTH_WARN failed to probe daemons or devices noout flag(s) set Reduced data availability: 339 pgs inactive services: mon: 1 daemons, quorum flucky-server (age 3m) mgr: flucky-server.cupbak(active, since 3m) mds: 1/1 daemons up osd: 18 osds: 18 up (since 26h), 18 in (since 7w) flags noout rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 339 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 339 unknown $ sudo ceph fs status cephfs - 2 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.flucky-server.ldzavv Reqs: 0 /s 61.9k 61.9k 17.1k 54.5k POOL TYPE USED AVAIL cephfs_metadata metadata 0 0 cephfs_data data 0 0 MDS version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) $ docker logs ceph-ddb7ebd8-65b5-11ed-84d7-22aca0408523-mon-flucky-server cluster 2023-01-27T12:15:30.437140+0000 mgr.flucky-server.cupbak (mgr.144098) 200 : cluster [DBG] pgmap v189: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:15:31.995+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:32.437854+0000 mgr.flucky-server.cupbak (mgr.144098) 201 : cluster [DBG] pgmap v190: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:32.373735+0000 osd.9 (osd.9) 123948 : cluster [DBG] 9.a deep-scrub starts cluster 2023-01-27T12:15:33.013990+0000 osd.2 (osd.2) 41797 : cluster [DBG] 5.6 scrub starts cluster 2023-01-27T12:15:33.402881+0000 osd.9 (osd.9) 123949 : cluster [DBG] 9.13 scrub starts cluster 2023-01-27T12:15:34.438591+0000 mgr.flucky-server.cupbak (mgr.144098) 202 : cluster [DBG] pgmap v191: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:35.461575+0000 osd.9 (osd.9) 123950 : cluster [DBG] 7.16 deep-scrub starts debug 2023-01-27T12:15:37.005+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:36.439416+0000 mgr.flucky-server.cupbak (mgr.144098) 203 : cluster [DBG] pgmap v192: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:36.925368+0000 osd.2 (osd.2) 41798 : cluster [DBG] 7.15 deep-scrub starts cluster 2023-01-27T12:15:37.960907+0000 osd.2 (osd.2) 41799 : cluster [DBG] 6.6 scrub starts cluster 2023-01-27T12:15:38.440099+0000 mgr.flucky-server.cupbak (mgr.144098) 204 : cluster [DBG] pgmap v193: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:38.482333+0000 osd.9 (osd.9) 123951 : cluster [DBG] 2.2 scrub starts cluster 2023-01-27T12:15:38.959557+0000 osd.2 (osd.2) 41800 : cluster [DBG] 9.47 scrub starts cluster 2023-01-27T12:15:39.519980+0000 osd.9 (osd.9) 123952 : cluster [DBG] 4.b scrub starts cluster 2023-01-27T12:15:40.440711+0000 mgr.flucky-server.cupbak (mgr.144098) 205 : cluster [DBG] pgmap v194: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:15:42.012+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:41.536421+0000 osd.9 (osd.9) 123953 : cluster [DBG] 2.7 scrub starts cluster 2023-01-27T12:15:42.441314+0000 mgr.flucky-server.cupbak (mgr.144098) 206 : cluster [DBG] pgmap v195: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:43.954128+0000 osd.2 (osd.2) 41801 : cluster [DBG] 9.4f scrub starts cluster 2023-01-27T12:15:44.441897+0000 mgr.flucky-server.cupbak (mgr.144098) 207 : cluster [DBG] pgmap v196: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:45.944038+0000 osd.2 (osd.2) 41802 : cluster [DBG] 1.1f deep-scrub starts debug 2023-01-27T12:15:47.019+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:46.442532+0000 mgr.flucky-server.cupbak (mgr.144098) 208 : cluster [DBG] pgmap v197: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:47.543275+0000 osd.9 (osd.9) 123954 : cluster [DBG] 2.3 scrub starts cluster 2023-01-27T12:15:48.443081+0000 mgr.flucky-server.cupbak (mgr.144098) 209 : cluster [DBG] pgmap v198: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:48.515994+0000 osd.9 (osd.9) 123955 : cluster [DBG] 1.19 scrub starts cluster 2023-01-27T12:15:49.957501+0000 osd.2 (osd.2) 41803 : cluster [DBG] 7.11 scrub starts cluster 2023-01-27T12:15:50.443740+0000 mgr.flucky-server.cupbak (mgr.144098) 210 : cluster [DBG] pgmap v199: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:50.473278+0000 osd.9 (osd.9) 123956 : cluster [DBG] 5.10 scrub starts debug 2023-01-27T12:15:52.026+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:15:51.506790+0000 osd.9 (osd.9) 123957 : cluster [DBG] 5.1b deep-scrub starts cluster 2023-01-27T12:15:51.957026+0000 osd.2 (osd.2) 41804 : cluster [DBG] 4.16 scrub starts cluster 2023-01-27T12:15:52.444197+0000 mgr.flucky-server.cupbak (mgr.144098) 211 : cluster [DBG] pgmap v200: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:52.939466+0000 osd.2 (osd.2) 41805 : cluster [DBG] 5.1c scrub starts cluster 2023-01-27T12:15:53.470511+0000 osd.9 (osd.9) 123958 : cluster [DBG] 8.8 scrub starts cluster 2023-01-27T12:15:53.916653+0000 osd.2 (osd.2) 41806 : cluster [DBG] 5.6 deep-scrub starts cluster 2023-01-27T12:15:54.422547+0000 osd.9 (osd.9) 123959 : cluster [DBG] 9.3b deep-scrub starts cluster 2023-01-27T12:15:54.444675+0000 mgr.flucky-server.cupbak (mgr.144098) 212 : cluster [DBG] pgmap v201: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:55.409322+0000 osd.9 (osd.9) 123960 : cluster [DBG] 9.34 deep-scrub starts cluster 2023-01-27T12:15:55.921989+0000 osd.2 (osd.2) 41807 : cluster [DBG] 7.15 deep-scrub starts debug 2023-01-27T12:15:57.029+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 audit 2023-01-27T12:15:56.339185+0000 mgr.flucky-server.cupbak (mgr.144098) 213 : audit [DBG] from='client.144120 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch cluster 2023-01-27T12:15:56.445186+0000 mgr.flucky-server.cupbak (mgr.144098) 214 : cluster [DBG] pgmap v202: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:57.883819+0000 osd.2 (osd.2) 41808 : cluster [DBG] 6.6 deep-scrub starts cluster 2023-01-27T12:15:58.445697+0000 mgr.flucky-server.cupbak (mgr.144098) 215 : cluster [DBG] pgmap v203: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:15:59.415908+0000 osd.9 (osd.9) 123961 : cluster [DBG] 9.25 scrub starts cluster 2023-01-27T12:16:00.446210+0000 mgr.flucky-server.cupbak (mgr.144098) 216 : cluster [DBG] pgmap v204: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:16:02.033+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:16:02.446670+0000 mgr.flucky-server.cupbak (mgr.144098) 217 : cluster [DBG] pgmap v205: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2023-01-27T12:16:04.953+0000 7fa908bf2700 0 mon.flucky-server@0(leader) e1 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"} v 0) v1 debug 2023-01-27T12:16:04.953+0000 7fa908bf2700 0 log_channel(audit) log [INF] : from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"}]: dispatch debug 2023-01-27T12:16:04.969+0000 7fa908bf2700 0 mon.flucky-server@0(leader) e1 handle_command mon_command({"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"} v 0) v1 debug 2023-01-27T12:16:04.969+0000 7fa908bf2700 0 log_channel(audit) log [INF] : from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"}]: dispatch cluster 2023-01-27T12:16:04.447207+0000 mgr.flucky-server.cupbak (mgr.144098) 218 : cluster [DBG] pgmap v206: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:16:04.537785+0000 osd.9 (osd.9) 123962 : cluster [DBG] 9.27 scrub starts cluster 2023-01-27T12:16:04.795757+0000 osd.2 (osd.2) 41809 : cluster [DBG] 9.47 scrub starts audit 2023-01-27T12:16:04.956941+0000 mon.flucky-server (mon.0) 304 : audit [INF] from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/mirror_snapshot_schedule"}]: dispatch audit 2023-01-27T12:16:04.973875+0000 mon.flucky-server (mon.0) 305 : audit [INF] from='mgr.144098 172.18.0.1:0/3192812764' entity='mgr.flucky-server.cupbak' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/flucky-server.cupbak/trash_purge_schedule"}]: dispatch debug 2023-01-27T12:16:07.039+0000 7fa90b3f7700 1 mon.flucky-server(a)0(leader).osd e50043 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 348127232 full_alloc: 348127232 kv_alloc: 322961408 cluster 2023-01-27T12:16:06.447964+0000 mgr.flucky-server.cupbak (mgr.144098) 219 : cluster [DBG] pgmap v207: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:16:07.606921+0000 osd.9 (osd.9) 123963 : cluster [DBG] 9.1c scrub starts cluster 2023-01-27T12:16:08.448450+0000 mgr.flucky-server.cupbak (mgr.144098) 220 : cluster [DBG] pgmap v208: 339 pgs: 339 unknown; 0 B data, 0 B used, 0 B / 0 B avail cluster 2023-01-27T12:16:08.629529+0000 osd.9 (osd.9) 123964 : cluster [DBG] 9.2c scrub starts

2 1

excluding from host_pattern
by E Taka 28 Jan '23

28 Jan '23

Hi, I wonder if it is possible to define a host pattern, which includes the host names ceph01…ceph19, but no other hosts, especially not ceph00. That means, this pattern is wrong: ceph[01][0-9] , since it includes ceph00. Not really a problem, but it seems that the "“host-pattern” is a regex that matches against hostnames and returns only matching hosts"¹ is not defined more precisely in the docs. 1) https://docs.ceph.com/en/latest/cephadm/host-management/

4 4

Re: Ceph rbd clients surrender exclusive lock in critical situation
by Marc 28 Jan '23

28 Jan '23

> > > > Hi all, > > > > we are observing a problem on a libvirt virtualisation cluster that > might come from ceph rbd clients. Something went wrong during execution > of a live-migration operation and as a result we have two instances of > the same VM running on 2 different hosts, the source- and the > destination host. What we observe now is the the exclusive lock of the > RBD disk image moves between these two clients periodically (every few > minutes the owner flips). > > Hi Frank, > > If you are talking about RBD exclusive lock feature ("exclusive-lock" > under "features" in "rbd info" output) then this is expected. This > feature provides automatic cooperative lock transitions between clients > to ensure that only a single client is writing to the image at any > given time. It's there to protect internal per-image data structures > such as the object map, the journal or the client-side PWL (persistent > write log) cache from concurrent modifications in case the image is > opened by two or more clients. The name is confusing but it's NOT > about preventing other clients from opening and writing to the image. > Rather it's about serializing those writes. > I can remember asking this also quite some time ago. Maybe this is helpful https://www.wogri.at/scripts/ceph-libvirt-locking/

4 6

rbd-mirror replication speed is very slow - but initial replication is fast
by ankit raikwar 28 Jan '23

28 Jan '23

Hello Team, Please help me i deploy two ceph cluster with 6 node configuration almost 800tb of capacity. and configurae in the DC-DR configuration for the data high availability. i eanbel the rwg and rbd block device mirroring for the replocatio of the data. we have the 10 GBPS fiber replication network . when we first start rbd mirror from our dc to dr starting time when we are replication our exsisting data that time we are getting almomst 8 GBPS replication speed and it's work fine. once all the exesting images data replicated now we are facing the replication speed issue . now only we are getting the 5 to 10 mbps relication speed. we also try to find the option like rbd_journal_max_payload_bytes and rbd_mirror_journal_max_fetch_bytes but max payload size we try to increase but we don't get any result regarding the speed. it still same . and rbd_mirror_journal_max_fetch_bytes option we are not able to find on the our ceph version. i also try to modify some other values and increase like rbd_mirror_memory_target rbd_mirror_memory_cache_min you also can find some refrence regarding this values for increase performace. Eugen [1] https://tracker.ceph.com/projects/ceph/repository/revisions/1ef12ea0d29f955… [2] https://github.com/ceph/ceph/pull/27670 Information of my ceph Cluster. Version: ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable) rbd-mirror daemon version: 17.2.5 Mirror mode; pool max image mirro at time: 5 replication network: 10 gbps (dedicated) Client: DC cluster we are continue writing the 50 to 400 mbps data but replication only 5 to 10 mbps. issue: speed only we get the 4 to 5 mbps. also we have the 10 Gbps replication network bandwidth. Note::- I also try to find the option rbd_mirror_journal_max_fetch_bytes but i'm not able to find the this option in the configuration. also when i try to set from the command line it's showing error like command: ceph config set client.rbd rbd_mirror_journal_max_fetch_bytes 33554432 error: Error EINVAL: unrecognized config option 'rbd_mirror_journal_max_fetch_bytes'

1 0

OSDs fail to start after stopping them with ceph osd stop command
by Stefan Hanreich 27 Jan '23

27 Jan '23

We encountered the following problems while trying to perform maintenance on a Ceph cluster: The cluster consists of 7 Nodes with 10 OSDs each. There are 4 pools on it: 3 of them are replicated pools with 3/2 size/min_size and one is an erasure coded pool with m=2 and k=5. The following global flags were set: * noout * norebalance * nobackfill * norecover Then, after those flags were set, all OSDs were stopped via the command ceph osd stop, which seems to have caused the issue. After maintenance was done, all OSDs were started again via systemctl. Only about half of the 70 OSDs in total started at first - while the other half started, but got killed after a few seconds with the following log messages: ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff3fcf8d700 -1 osd.51 12161 map says i am stopped by admin. shutting down. ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff40da55700 -1 received signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff40da55700 -1 osd.51 12161 *** Got signal Interrupt *** ceph-osd[197270]: 2023-01-24T13:39:12.103+0100 7ff40da55700 -1 osd.51 12161 *** Immediate shutdown (osd_fast_shutdown=true) *** And indeed, when looking into the osd map via ceph osd dump, the remaining OSDs seem to be marked as stopped: osd.50 down out weight 0 up_from 9213 up_thru 9416 down_at 9760 last_clean_interval [9106,9207) [v2:10.0.1.61:6813/6211,v1:10.0.1.61:6818/6211] [v2:10.0.0.61:6814/6211,v1:10.0.0.61:6816/6211] exists,stop 9a2590c4-f50b-4550-bfd1-5aafb543cb59 We were able to restore some of the remaining OSDs via running ceph out osd XX ceph in osd XX and then starting the service again (via systemctl start). This did work for most OSDs, except for the OSDs that are located on one specific host. Some OSDs required several restarts until they did not kill themselves a few seconds after starting. This whole issue seems to be caused by the OSDs being marked as stopped in the OSD map [1]. Apparently this state should get reset when re-starting the OSD again [2], but for some reason this doesn't happen for some of the OSDs. This behavior seems to have been introduced via the following pull request [3]. We have also found the following commit where the logic regarding stop seemed to have been introduced [4]. We were looking into commands that reset the stopped status of the OSD in the OSD map, but did not find any way of forcing this. Since we are out of ideas on how to proceed with the remaining 10 OSDs that cannot get brought up: How does one recover from this situation? It seems like by running ceph osd stop the cluster got in a state that seems irrecoverable with the normal CLI commands available. We even looked into the possibility of manually manipulating the osdmap via the osdmaptool, but there doesn't seem to be a way to edit the start/stopped status and it also seems like a very invasive procedure. There does not seem to be any way we can see of recovering from this, apart from rebuilding all the OSDs - which we refrained from for now. Kind Regards Hanreich Stefan [1] https://github.com/ceph/ceph/blob/63a77b2c5b683cb241f865daec92c046152175b4/… [2] https://github.com/ceph/ceph/blob/63a77b2c5b683cb241f865daec92c046152175b4/… [3] https://github.com/ceph/ceph/pull/43664 [4] https://github.com/ceph/ceph/commit/5dbae13ce0f5b0104ab43e0ccfe94f832d0e1268

2 2

Replacing OSD with containerized deployment
by Ken D 27 Jan '23

27 Jan '23

Dear Ceph-Users, i am struggling to replace a disk. My ceph-cluster is not replacing the old OSD even though I did: ceph orch osd rm 232 --replace The OSD 232 is still shown in the osd list, but the new hdd will be placed as a new OSD. This wouldnt mind me much, if the OSD was also placed on the bluestoreDB / NVME, but it doesn't. My steps: "ceph orch osd rm 232 --replace" remove the failed hdd. add the new one. Convert the disk within the servers bios, so that the node can have direct access on it. It shows up as /dev/sdt, enter maintenance mode reboot server drive is now /dev/sdm (which the old drive had) "ceph orch device zap node-x /dev/sdm " A new OSD is placed on the cluster. Can you give me a hint, where did I take a wrong turn? Why is the disk not being used as OSD 232? Best Ken P.S. Sorry for double sending this message, somehow this mail-address was not subscribed to the list anymore.

1 0

Replacing OSD with containerized deployment
by mailing-lists 27 Jan '23

27 Jan '23

Dear Ceph-Users, i am struggling to replace a disk. My ceph-cluster is not replacing the old OSD even though I did: ceph orch osd rm 232 --replace The OSD 232 is still shown in the osd list, but the new hdd will be placed as a new OSD. This wouldnt mind me much, if the OSD was also placed on the bluestoreDB / NVME, but it doesn't. My steps: "ceph orch osd rm 232 --replace" remove the failed hdd. add the new one. Convert the disk within the servers bios, so that the node can have direct access on it. It shows up as /dev/sdt, enter maintenance mode reboot server drive is now /dev/sdm (which the old drive had) "ceph orch device zap node-x /dev/sdm " A new OSD is placed on the cluster. Can you give me a hint, where did I take a wrong turn? Why is the disk not being used as OSD 232? Best Ken P.S. Sorry for double sending this message, somehow this mail was not subscribed to the list anymore.

1 0

Audit logs of creating RBD volumes and creating RGW buckets
by Jinhao Hu 27 Jan '23

27 Jan '23

Hi, Are the creation of RBD volumes and RGW buckets audited? If yes, what do the audit logs look like? Is there any documentation about it? I tried to find the related audit logs from the "/var/log/ceph/ceph.audit.log" file but didn't find any. Thanks, Jinhao

1 0

Re: OSDs are not utilized evenly
by Jeremy Austin 27 Jan '23

27 Jan '23

Are there alternatives to TheJJ balancer? I have a (temporary) rebalance problem, and that code chokes[1]. Essentially, I have a few pgs in remapped+backfill_toofull, but plenty of space in the parent's parent bucket(s). [1] https://github.com/TheJJ/ceph-balancer/issues/23 On Wed, Dec 14, 2022 at 6:55 AM Denis Polom <denispolom(a)gmail.com> wrote: > Hi, > > looks like TheJJ balancer solved the issue! > > Thx! > > > On 11/9/22 13:35, Denis Polom wrote: > > Hi Stefan, > > > > thank you for help. Looks very interesting and command you sent helps > > to have better insight on that. Still wandering why some of OSDs keeps > > primary for more PGs as others. I was thinking that balancer and CRUSH > > should take care of that. > > > > I will try balancer you sent a link for and will post result. But this > > will take more time as first I have to test it on some non-production > > Ceph. > > > > Thx! > > > > > > On 11/9/22 08:20, Stefan Kooman wrote: > >> On 11/1/22 13:45, Denis Polom wrote: > >>> Hi > >>> > >>> I observed on my Ceph cluster running latest Pacific that same size > >>> OSDs are utilized differently even if balancer is running and > >>> reports status as perfectly balanced. > >>> > >> > >> That might be true because the primary PGs are not evenly balanced. > >> You can check that with: ceph pg dump. The last output is the > >> overview for how many PGs an OSD is primary for. To get more detail > >> by pool you can run this (source: unknown, but it works :-)): > >> > >> "ceph pg dump | awk ' > >> BEGIN { IGNORECASE = 1 } > >> /^PG_STAT/ { col=1; while($col!="UP") {col++}; col++ } > >> /^[0-9a-f]+\.[0-9a-f]+/ { match($0,/^[0-9a-f]+/); pool=substr($0, > >> RSTART, RLENGTH); poollist[pool]=0; > >> up=$col; i=0; RSTART=0; RLENGTH=0; delete osds; > >> while(match(up,/[0-9]+/)>0) { osds[++i]=substr(up,RSTART,RLENGTH); up > >> = substr(up, RSTART+RLENGTH) } > >> for(i in osds) {array[osds[i],pool]++; osdlist[osds[i]];} > >> } > >> END { > >> printf("\n"); > >> printf("pool :\t"); for (i in poollist) printf("%s\t",i); printf("| > >> SUM \n"); > >> for (i in poollist) printf("--------"); printf("----------------\n"); > >> for (i in osdlist) { printf("osd.%i\t", i); sum=0; > >> for (j in poollist) { printf("%i\t", array[i,j]); sum+=array[i,j]; > >> sumpool[j]+=array[i,j] }; printf("| %i\n",sum) } > >> for (i in poollist) printf("--------"); printf("----------------\n"); > >> printf("SUM :\t"); for (i in poollist) printf("%s\t",sumpool[i]); > >> printf("|\n"); > >> }'" > >> > >> 11/15/2022 14:35 UTC there is a talk about this: New workload > >> balancer in Ceph (Ceph virtual 2022). > >> > >> The balancer made by Jonas Jelten works very well for us (though does > >> not balance primary PGs): https://github.com/TheJJ/ceph-balancer. It > >> outperforms the ceph-balancer module by far. And had faster > >> convergence. This is true up to and including octopus release. > >> > >> Gr. Stefan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Jeremy Austin jhaustin(a)gmail.com

2 2

ceph 16.2.10 cluster down
by Jens Galsgaard 27 Jan '23

27 Jan '23

Hello, I'm currently investigating a downed ceph cluster that I cannot communicate with. Setup is: 3 hosts with each 12 disks (osd/mon) 3 vm's with mon/mds/mgr The vm's are unavailable at the moment and one of the hosts is online with osd/mon running. When issuing the command ceph -s nothing happens and after 5 minutes the following. 2023-01-26T12:40:08.111+0100 7f68b4b8b700 0 monclient(hunting): authenticate timed out after 300 What would be the best way of troubleshooting this? Venlig hilsen - Mit freundlichen Grüßen - Kind Regards, Jens Galsgaard

2 4

Ceph Days Co-Located with SCALE - CFP ends in 1 week
by Mike Perez 27 Jan '23

27 Jan '23

Hi everyone, Ceph Days are coming to Southern California, co-located with our friends at SCALE - the Southern California Linux Expo! The event will be a full day of Ceph content on March 9. The CFP ends on *2023-02-02*, so start drafting and completing your proposals quickly. https://survey.zohopublic.com/zs/3UBUpC https://ceph.io/en/community/events/2023/ceph-days-socal/ Here are some suggested topics: - Ceph operations, management, and development - New and proposed Ceph features, development status - Ceph development roadmap - Best practices - Ceph use-cases, solution architectures, and user experiences - Ceph performance and optimization - Platform Integrations - Kubernetes, OpenShift - OpenStack (Cinder, Manila, etc.) - Spark - Multi-site and multi-cluster data services - Persistent memory, ZNS SSDs, SMR HDDs, DPUs, and other new hardware technologies - Storage management, monitoring, and deployment automation - Experiences deploying and operating Ceph in production and/or at scale - Small-scale or edge deployments - Long-term, archival storage - Data compression, deduplication, and storage optimization - Developer processes, tools, challenges - Ceph testing infrastructure, tools - Ceph community issues, outreach, and project governance - Ceph documentation, training, and learner experience -- Mike Perez There are other Ceph speaking opportunities to consider: - Ceph Days NYC <https://ceph.io/en/community/events/2023/ceph-days-nyc/> - February 21st, 2023 - Schedule and registration are available - Ceph Days Southern California <https://ceph.io/en/community/events/2023/ceph-days-socal/> - March 9th, 2023 - CFP open until *February 2nd*. - Cephalocon 2023 <https://events.linuxfoundation.org/cephalocon/> (co-located with KubeCon in Amsterdam) - April 16 - 18 - CFP now available! - Ceph Tech Talk (virtual) <https://ceph.io/en/community/tech-talks/> - Monthly Make sure to join our Announcement list or social media for further updates on events - Ceph Announcement list <https://lists.ceph.io/postorius/lists/ceph-announce.ceph.io/> - Twitter <https://twitter.com/ceph> - LinkedIn <https://www.linkedin.com/company/ceph/> - FaceBook <https://www.facebook.com/cephstorage/>

1 0

Cannot delete images in rbd_trash
by Nikhil Shah 26 Jan '23

26 Jan '23

Hello, We have a couple of RBD images in a pool that are unable to be deleted. The user attempted to delete these volumes , while we were in the middle of a ceph minor version upgrade (where ceph processes restart). I am suspecting that during one of the service restarts (probably monitor?), the image deletion only got half way, and let these images in a bad state. It looks like the images were scheduled to move to the trash (from the rbd rm), but did not make it. Their omap values still exist in the trash though. Any suggestions on how these can be cleaned up? We are running Ceph 15.2.17 (now , after the upgrade from Ceph 15.2.15/16). Thank you #rbd rm 418824e0-576a-4167-957a-5f3fa6f2693a -p <pool name> Removing image: 0% complete...failed. rbd: delete error: (117) Structure needs cleaning 2023-01-26T15:14:40.258+0000 7f2677dfb380 -1 librbd::api::Trash: remove: error: image is pending moving to the trash. #rbd -p <pool name> trash ls no output #rbd -p <pool> ls | grep -i 418824e0-576a-4167-957a-5f3fa6f2693a 418824e0-576a-4167-957a-5f3fa6f2693a #rbd info <pool name>/418824e0-576a-4167-957a-5f3fa6f2693a rbd: error opening image 418824e0-576a-4167-957a-5f3fa6f2693a: (2) No such file or directory #rados -p <pool name> listomapvals rbd_trash id_db434b7d01af76 value (64 bytes) : 00000000 02 01 3a 00 00 00 03 24 00 00 00 34 31 38 38 32 |..:....$...41882| 00000010 34 65 30 2d 35 37 36 61 2d 34 31 36 37 2d 39 35 |4e0-576a-4167-95| 00000020 37 61 2d 35 66 33 66 61 36 66 32 36 39 33 61 0e |7a-5f3fa6f2693a.| 00000030 41 22 63 06 f7 bf 23 0e 41 22 63 06 f7 bf 23 01 |A"c...#.A"c...#.| 00000040 id_db5431429f149d value (64 bytes) : 00000000 02 01 3a 00 00 00 03 24 00 00 00 33 65 31 63 65 |..:....$...3e1ce| 00000010 61 34 34 2d 30 31 63 35 2d 34 62 65 39 2d 39 31 |a44-01c5-4be9-91| 00000020 32 30 2d 37 65 64 64 62 66 37 65 39 65 32 63 0e |20-7eddbf7e9e2c.| 00000030 41 22 63 97 2b 3e 20 0e 41 22 63 97 2b 3e 20 01 |A"c.+> .A"c.+> .| 00000040 Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at legal(a)kaseya.com, and then delete it. To opt-out of receiving emails Please click here<https://info.kaseya.com/email-subscription-center.html>. The term 'this e-mail' includes any and all attachments.

1 0

Octopus mgr doesn't resume after boot
by Renata Callado Borges 26 Jan '23

26 Jan '23

Dear all, I have a two hosts setup, and I recently rebooted a mgr machine without "set noout" and "set norebalance" commands. The "darkside2" machine is the cephadm machine, and "darkside3" is the improperly rebooted mgr. Now the darkside3 machine does not resume ceph configuration: [root@darkside2 ~]# ceph orch host ls HOST ADDR LABELS STATUS darkside2 darkside2 darkside3 172.22.132.189 Offline If I understood the docs correctly, I should [root@darkside2 ~]# ceph orch host add darkside3 But this fails because darkside3 doesn't accept root ssh connnections. I presume this has been discussed before, but I couldn't find the correct thread. Could someone please point me in the right direction? Cordially, Renata.

1 1

Debian update to 16.2.11-1~bpo11+1 failing
by Luke Hall 26 Jan '23

26 Jan '23

Hi, Trying to dist-upgrade an osd server this morning and lots of necessary packages have been removed! Start-Date: 2023-01-26 10:04:57 Commandline: apt dist-upgrade Install: linux-image-5.10.0-21-amd64:amd64 (5.10.162-1, automatic) Upgrade: librados2:amd64 (16.2.10-1~bpo11+1, 16.2.11-1~bpo11+1), ceph-mgr-modules-core:amd64 (16.2.10-1~bpo11+1, 16.2.11-1~bpo11+1), librbd1:amd64 (16.2.10-1~bpo11+1, 16.2.11-1~bpo11+1), librgw2:amd64 (16.2.10-1~bpo11+1, 16.2.11-1~bpo11+1), linux-image-amd64:amd64 (5.10.158-2, 5.10.162-1), libradosstriper1:amd64 (16.2.10-1~bpo11+1, 16.2.11-1~bpo11+1), cephadm:amd64 (16.2.10-1~bpo11+1, 16.2.11-1~bpo11+1), linux-libc-dev:amd64 (5.10.158-2, 5.10.162-1) Remove: ceph-base:amd64 (16.2.10-1~bpo11+1), linux-image-5.10.0-15-amd64:amd64 (5.10.120-1), ceph-mgr-cephadm:amd64 (16.2.10-1~bpo11+1), ceph-common:amd64 (16.2.10-1~bpo11+1), radosgw:amd64 (16.2.10-1~bpo11+1), linux-image-5.10.0-17-amd64:amd64 (5.10.136-1), ceph-mds:amd64 (16.2.10-1~bpo11+1), ceph-mgr:amd64 (16.2.10-1~bpo11+1), ceph-mon:amd64 (16.2.10-1~bpo11+1), ceph-osd:amd64 (16.2.10-1~bpo11+1), ceph-mgr-diskprediction-local:amd64 (16.2.10-1~bpo11+1), ceph:amd64 (16.2.10-1~bpo11+1), ceph-mgr-dashboard:amd64 (16.2.10-1~bpo11+1), ceph-mgr-k8sevents:amd64 (16.2.10-1~bpo11+1) End-Date: 2023-01-26 10:06:26 Seems ceph-common depends on packages which aren't (yet) in the repo The following packages have unmet dependencies: ceph-common : Depends: python3-cephfs (= 16.2.11-1~bpo11+1) but it is not going to be installed Depends: python3-rados (= 16.2.11-1~bpo11+1) but 16.2.10-1~bpo11+1 is to be installed Depends: python3-rbd (= 16.2.11-1~bpo11+1) but 16.2.10-1~bpo11+1 is to be installed Depends: python3-rgw (= 16.2.11-1~bpo11+1) but 16.2.10-1~bpo11+1 is to be installed E: Unable to correct problems, you have held broken packages. Is anyone aware of this? Thanks -- All postal correspondence to: The Positive Internet Company, 24 Ganton Street, London. W1F 7QY *Follow us on Twitter* @posipeople The Positive Internet Company Limited is registered in England and Wales. Registered company number: 3673639. VAT no: 726 7072 28. Registered office: Northside House, Mount Pleasant, Barnet, Herts, EN4 9EE.

1 2

Cephalocon 2023 Is Coming to Amsterdam! CFP Is Now Open!
by Mike Perez 26 Jan '23

26 Jan '23

Cephalocon Is Coming to Amsterdam‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ [image: Logo] https://events.linuxfoundation.org/cephalocon/[image: Cephalocon April 16-18, 2023 in Amsterdam, Netherlands] Cephalocon Is Coming to Amsterdam Cephalocon 2023 <https://events.linuxfoundation.org/cephalocon> is coming to Amsterdam on April 16-18, co-located with KubeCon! Cephalocon is the premier yearly event that brings together the global community of operators, developers, and researchers for Ceph, the open source distributed storage system designed to provide excellent performance, reliability, and scalability. Join new and existing community members from around the world to learn more about Ceph and the future of the project from the developers writing the code and the operators deploying it at scale. The CFP is now open <https://events.linuxfoundation.org/cephalocon/program/cfp/>, and registration will be available coming soon! Here are some important dates: - *CFP Closes:* Sunday, February 12 at 11:59 pm PST - *CFP Notifications:* Wednesday, February 22 - *Schedule Announcement:* Monday, February 27 - *Presentation Slide Due Date: *Wednesday, April 12 - *Event Dates: *Monday, April 17 – Tuesday, April 18 (Developer Summit Sunday, April 16) The sponsorship prospectus is available <https://events.linuxfoundation.org/wp-content/uploads/2023/01/sponsor-ceph-…>. Contact sponsorships(a)ceph.foundation to secure your sponsorship, request additional details, or discuss custom options. Thank you to our event planners and everyone who has contributed to planning discussions. We look forward to seeing you all soon! ------ There are other Ceph speaking opportunities to consider: - Ceph Days NYC <https://ceph.io/en/community/events/2023/ceph-days-nyc/> - February 21st, 2023 - Schedule and registration are available - Ceph Days Southern California <https://ceph.io/en/community/events/2023/ceph-days-socal/> - March 9th, 2023 - CFP open until *February 2nd*. - Ceph Tech Talk (virtual) <https://ceph.io/en/community/tech-talks/> - Monthly SUBMIT YOUR PROPOSAL <https://events.linuxfoundation.org/cephalocon/> [image: Link icon] <https://ceph.io> [image: Twitter icon] <https://twitter.com/ceph> [image: YouTube icon] <https://youtube.com/c/cephstorage> [image: LinkedIn icon] <https://www.linkedin.com/company/ceph/> [image: Facebook icon] <https://www.facebook.com/cephstorage/> jee74vonjgtz

1 0

v16.2.11 Pacific released
by Yuri Weinstein 26 Jan '23

26 Jan '23

We're happy to announce the 11th backport release in the Pacific series. We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2023/v16-2-11-pacific-released Notable Changes --------------- * Cephfs: The 'AT_NO_ATTR_SYNC' macro is deprecated, please use the standard 'AT_STATX_DONT_SYNC' macro. The 'AT_NO_ATTR_SYNC' macro will be removed in the future. * Trimming of PGLog dups is now controlled by the size instead of the version. This fixes the PGLog inflation issue that was happening when the on-line (in OSD) trimming got jammed after a PG split operation. Also, a new off-line mechanism has been added: `ceph-objectstore-tool` got `trim-pg-log-dups` op that targets situations where OSD is unable to boot due to those inflated dups. If that is the case, in OSD logs the "You can be hit by THE DUPS BUG" warning will be visible. Relevant tracker: https://tracker.ceph.com/issues/53729 * RBD: `rbd device unmap` command gained `--namespace` option. Support for namespaces was added to RBD in Nautilus 14.2.0 and it has been possible to map and unmap images in namespaces using the `image-spec` syntax since then but the corresponding option available in most other commands was missing. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.11.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 3cf40e2dca667f68c6ce3ff5cd94f01e711af894

1 0

Mount ceph using FQDN
by kushagra.gupta＠hsc.com 25 Jan '23

25 Jan '23

Hi team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 We have a dns server with ip abcd:abcd:abcd::31 which resolves the above ip's with a single hostname. The resolution is as follows: ``` $TTL 1D @ IN SOA storage.com root ( 6 ; serial 1D ; refresh 1H ; retry 1W ; expire 3H ) ; minimum IN NS master master IN A 10.0.1.31 storagenode IN AAAA abcd:abcd:abcd::21 storagenode IN AAAA abcd:abcd:abcd::22 storagenode IN AAAA abcd:abcd:abcd::23 ``` We want to mount the ceph storage on a node using this hostname. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip 'storagenode.storage.com:6789' ``` We also tried mounting ceph storage by removing the dns server and resolving the ip as follows: ``` abcd:abcd:abcd::21 storagenode1 ``` But we are getting similar results. Also kindly note that we are able to perform the mount operation if we use ips instead of domain name. Could you please help us out with how we can mount ceph using FQDN. Kindly let me know if any other imformation is required. My ceph.conf configuration is as follows: ``` [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 ``` Thanks and Regards

3 2

Image corrupt after restoring snapshot via Proxmox
by Roel van Meer 25 Jan '23

25 Jan '23

Hi, We have had a situation three times where rbd images seem to be corrupt after restoring a snapshot, and I'm looking for advice on how to investigate this. We're running Proxmox 7 with Ceph Octopus (Proxmox build, 15.2.17-pve1). Every time the problem has happened, it has happened after these actions were done with the VM: (Yesterday) - VM stopped - Snapshot created - VM started - VM stopped - Snapshot restored - VM started (OK) - Nightly backup with vzdump to Proxmox Backup Server (Today) - VM stopped - Snapshot restored - VM does not start On previous occasions we tried to find a solution and when we couldn't, we restored the VM from backup, which solved the problem. Now this happened to a test system, so we've left the situation as is and maybe get to the root cause. Some observations: * We're using krbd * The PBS backups don't allow file restore if the backup was made from a "broken" image * After mapping the current image, it doesn't seem to contain a partition table There's a thread on the Proxmox forum about this issues as well[1]. If anyone could give some advice about how to proceed from here, I'd be very grateful. Best regards, Roel PS: An upgrade to Pacific has already been planned. [1] https://forum.proxmox.com/threads/vm-disks-corrupt-after-reverting-to-snaps… -- Wij zijn ISO 27001 gecertificeerd 1A First Alternative BV T: +31 (0)88 0016405 W: https://www.1afa.com

2 1

ceph cluster iops low
by petersun＠raksmart.com 25 Jan '23

25 Jan '23

I have my ceph IOPS very low with over 48 SSD backed on NVMs for DB/WAL on four physical servers. The whole cluster has only about 20K IO total. Looks the IOs are suppressed over bottleneck somewhere. Dstat shows a lots csw and interrupts over 150K, while I am using FIO bench 4K 128QD test. I check SSD throughput only about 40M at 250 ios each. Network are total 20G and not full of traffic. CPU are around 50% idle on 2*E5 2950v2 each node. Is it normal to get that high and how to reduce it? where else could be the bottleneck?

3 3

Re: 16.2.11 pacific QE validation status
by Yuri Weinstein 25 Jan '23

25 Jan '23

Happy New Year all! This release remains to be in "progress"/"on hold" status as we are sorting all infrastructure-related issues. Unless I hear objections, I suggest doing a full rebase/retest QE cycle (adding PRs merged lately) since it's taking much longer than anticipated when sepia is back online. Objections? Thx YuriW On Thu, Dec 15, 2022 at 9:14 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/58257#note-1 > Release Notes - TBD > > Seeking approvals for: > > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still being > tested and will be merged soon) > rook - Sébastien Han > cephadm - Adam > dashboard - Ernesto > rgw - Casey (rwg will be rerun on the latest SHA1) > rbd - Ilya, Deepika > krbd - Ilya, Deepika > fs - Venky, Patrick > upgrade/nautilus-x (pacific) - Neha, Laura > upgrade/octopus-x (pacific) - Neha, Laura > upgrade/pacific-p2p - Neha - Neha, Laura > powercycle - Brad > ceph-volume - Guillaume, Adam K > > Thx > YuriW

12 21

Integrating openstack/swift to ceph cluster
by Michel Niyoyita 24 Jan '23

24 Jan '23

Hello team,, I have deployed ceph pacific cluster using ceph-ansible running on ubuntu 20.04 which have 3 OSD hosts and 3 mons on each OSD host we have 20 osd . I am integrating swift in the cluster but I fail to find the policy and upload objects in the container . I have deployed rgwloadbalancer on top below is my ceph.conf configuration and rgw logs. Kindly help if I am missing configs in my ceph.conf and advise. [client] rbd_default_features = 1 [client.rgw.ceph-osd1] rgw_dns_name = ceph-osd1 [client.rgw.ceph-osd1.rgw0] host = ceph-osd1 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd1.rgw0.log rgw frontends = beast endpoint=10.10.13.13:8080 rgw thread pool size = 512 rgw_dns_name = ceph-osd1 rgw_frontends = "beast port=8080" rgw_enable_usage_log = true rgw_thread_pool_size = 512 rgw_keystone_api_version = 3 rgw_keystone_url = http://10.10.13.31:5000 rgw_keystone_admin_user = admin rgw_keystone_admin_password = WyB9v1GPAqtxsySrCjZpa1L2U2JmYVF6zviP6bKk rgw_keystone_admin_domain = default rgw_keystone_admin_project = admin rgw_keystone_accepted_roles = admin,Member,_member_,member rgw_keystone_verify_ssl = false rgw_s3_auth_use_keystone = true [client.rgw.ceph-osd2] rgw_dns_name = ceph-osd2 [client.rgw.ceph-osd2.rgw0] host = ceph-osd2 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd2.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd2.rgw0.log rgw frontends = beast endpoint=10.10.13.14:8080 rgw thread pool size = 512 rgw_dns_name = ceph-osd2 rgw_frontends = "beast port=8080" rgw_enable_usage_log = true rgw_thread_pool_size = 512 rgw_keystone_api_version = 3 rgw_keystone_url = http://10.10.13.31:5000 rgw_keystone_admin_user = admin rgw_keystone_admin_password = WyB9v1GPAqtxsySrCjZpa1L2U2JmYVF6zviP6bKk rgw_keystone_admin_domain = default rgw_keystone_admin_project = admin rgw_keystone_accepted_roles = admin,Member,_member_,member rgw_keystone_verify_ssl = false rgw_s3_auth_use_keystone = true [client.rgw.ceph-osd3] rgw_dns_name = ceph-osd3 [client.rgw.ceph-osd3.rgw0] host = ceph-osd3 keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-osd3.rgw0/keyring log file = /var/log/ceph/ceph-rgw-ceph-osd3.rgw0.log rgw frontends = beast endpoint=10.10.13.15:8080 rgw thread pool size = 512 rgw_dns_name = ceph-osd3 rgw_frontends = "beast port=8080" rgw_enable_usage_log = true rgw_thread_pool_size = 512 rgw_keystone_api_version = 3 rgw_keystone_url = http://10.10.13.31:5000 rgw_keystone_admin_user = admin rgw_keystone_admin_password = WyB9v1GPAqtxsySrCjZpa1L2U2JmYVF6zviP6bKk rgw_keystone_admin_domain = default rgw_keystone_admin_project = admin rgw_keystone_accepted_roles = admin,Member,_member_,member rgw_keystone_verify_ssl = false rgw_s3_auth_use_keystone = true # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster network = 10.10.13.0/24 fsid = c8ffac77-6e6e-479a-92be-08f9b46453e7 mon host = [v2:10.10.13.10:3300,v1:10.10.13.10:6789],[v2:10.10.13.11:3300 ,v1:10.10.13.11:6789],[v2:10.10.13.12:3300,v1:10.10.13.12:6789] mon initial members = ceph-mon1,ceph-mon2,ceph-mon3 mon_allow_pool_delete = True mon_max_pg_per_osd = 400 osd pool default crush rule = -1 osd_pool_default_min_size = 2 osd_pool_default_size = 3 public network = 0.0.0.0/0 2023-01-24T15:06:33.035+0000 7f0b4fe7f700 1 beast: 0x7f0d1055c6e0: 10.10.13.15 - anonymous [24/Jan/2023:15:06:33.035 +0000] "HEAD / HTTP/1.0" 200 5 - - - latency=0.000000000s 2023-01-24T15:06:33.855+0000 7f0b50e81700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:33.855+0000 7f0b50e81700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:33.855+0000 7f0b50e81700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:33.855+0000 7f0b50e81700 1 beast: 0x7f0d1055c6e0: 10.10.13.13 - anonymous [24/Jan/2023:15:06:33.855 +0000] "HEAD / HTTP/1.0" 200 0 - - - latency=0.000000000s 2023-01-24T15:06:33.907+0000 7f0b41e63700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:33.907+0000 7f0b42664700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:33.907+0000 7f0b42664700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:33.907+0000 7f0b42664700 1 beast: 0x7f0d1055c6e0: 10.10.13.14 - anonymous [24/Jan/2023:15:06:33.907 +0000] "HEAD / HTTP/1.0" 200 0 - - - latency=0.000000000s 2023-01-24T15:06:35.039+0000 7f0b37e4f700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:35.039+0000 7f0b37e4f700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:35.039+0000 7f0b37e4f700 1 beast: 0x7f0d1055c6e0: 10.10.13.15 - anonymous [24/Jan/2023:15:06:35.039 +0000] "HEAD / HTTP/1.0" 200 5 - - - latency=0.000000000s 2023-01-24T15:06:35.855+0000 7f0b32644700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:35.855+0000 7f0b32644700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:35.855+0000 7f0b32644700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:35.855+0000 7f0b32644700 1 beast: 0x7f0d1055c6e0: 10.10.13.13 - anonymous [24/Jan/2023:15:06:35.855 +0000] "HEAD / HTTP/1.0" 200 0 - - - latency=0.000000000s 2023-01-24T15:06:35.911+0000 7f0b31642700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:35.911+0000 7f0b31642700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:35.911+0000 7f0b31642700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:35.911+0000 7f0b31642700 1 beast: 0x7f0d1055c6e0: 10.10.13.14 - anonymous [24/Jan/2023:15:06:35.911 +0000] "HEAD / HTTP/1.0" 200 0 - - - latency=0.000000000s 2023-01-24T15:06:37.039+0000 7f0b2fe3f700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:37.043+0000 7f0b2fe3f700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.003999876s ====== 2023-01-24T15:06:37.043+0000 7f0b2fe3f700 1 beast: 0x7f0d1055c6e0: 10.10.13.15 - anonymous [24/Jan/2023:15:06:37.039 +0000] "HEAD / HTTP/1.0" 200 5 - - - latency=0.003999876s 2023-01-24T15:06:37.859+0000 7f0b28630700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:37.859+0000 7f0b28630700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:37.859+0000 7f0b28630700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:37.859+0000 7f0b28630700 1 beast: 0x7f0d1055c6e0: 10.10.13.13 - anonymous [24/Jan/2023:15:06:37.859 +0000] "HEAD / HTTP/1.0" 200 0 - - - latency=0.000000000s 2023-01-24T15:06:37.911+0000 7f0b1be17700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:37.911+0000 7f0b1be17700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:37.911+0000 7f0b1be17700 1 beast: 0x7f0d1055c6e0: 10.10.13.14 - anonymous [24/Jan/2023:15:06:37.911 +0000] "HEAD / HTTP/1.0" 200 5 - - - latency=0.000000000s 2023-01-24T15:06:39.043+0000 7f0b15e0b700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:39.043+0000 7f0b15e0b700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:39.043+0000 7f0b15e0b700 1 beast: 0x7f0d1055c6e0: 10.10.13.15 - anonymous [24/Jan/2023:15:06:39.043 +0000] "HEAD / HTTP/1.0" 200 5 - - - latency=0.000000000s 2023-01-24T15:06:39.859+0000 7f0b13e07700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:39.859+0000 7f0b13e07700 0 ERROR: client_io->complete_request() returned Connection reset by peer 2023-01-24T15:06:39.859+0000 7f0b13e07700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:39.859+0000 7f0b13e07700 1 beast: 0x7f0d1055c6e0: 10.10.13.13 - anonymous [24/Jan/2023:15:06:39.859 +0000] "HEAD / HTTP/1.0" 200 0 - - - latency=0.000000000s 2023-01-24T15:06:39.915+0000 7f0b12604700 1 ====== starting new request req=0x7f0d1055c6e0 ===== 2023-01-24T15:06:39.915+0000 7f0b12604700 1 ====== req done req=0x7f0d1055c6e0 op status=0 http_status=200 latency=0.000000000s ====== 2023-01-24T15:06:39.915+0000 7f0b12604700 1 beast: 0x7f0d1055c6e0: 10.10.13.14 - anonymous [24/Jan/2023:15:06:39.915 +0000] "HEAD / HTTP/1.0" 200 5 - - - latency=0.000000000s Best Regards Michel

1 0

Mds crash at cscs
by Lo Re Giuseppe 24 Jan '23

24 Jan '23

Dear all, We have started to use more intensively cephfs for some wlcg related workload. We have 3 active mds instances spread on 3 servers, mds_cache_memory_limit=12G, most of the other configs are default ones. One of them has crashed this night leaving the log below. Do you have any hint on what could be the cause and how to avoid it? Regards, Giuseppe [root@naret-monitor03 ~]# journalctl -u ceph-63334166-d991-11eb-99de-40a6b72108d0(a)mds.cephfs.naret-monitor03.lqppte.service ... Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific > Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fe291e4fce0] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 2: abort() Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 3: /lib64/libstdc++.so.6(+0x987ba) [0x7fe2912567ba] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 4: /lib64/libstdc++.so.6(+0x9653c) [0x7fe29125453c] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 5: /lib64/libstdc++.so.6(+0x95559) [0x7fe291253559] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 6: __gxx_personality_v0() Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 7: /lib64/libgcc_s.so.1(+0x10b03) [0x7fe290c34b03] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 8: _Unwind_Resume() Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 9: /usr/bin/ceph-mds(+0x18c104) [0x5638351e7104] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 10: /lib64/libpthread.so.0(+0x12ce0) [0x7fe291e4fce0] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 11: gsignal() Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 12: abort() Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 13: /lib64/libstdc++.so.6(+0x9009b) [0x7fe29124e09b] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 14: /lib64/libstdc++.so.6(+0x9653c) [0x7fe29125453c] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 15: /lib64/libstdc++.so.6(+0x96597) [0x7fe291254597] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 16: /lib64/libstdc++.so.6(+0x967f8) [0x7fe2912547f8] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 17: /lib64/libtcmalloc.so.4(+0x19fa4) [0x7fe29bae6fa4] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 18: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, vo> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 19: (std::shared_ptr<inode_t<mempool::mds_co::pool_allocator> > InodeSt> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 20: (CInode::_decode_base(ceph::buffer::v15_2_0::list::iterator_impl<tr> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 21: (CInode::decode_import(ceph::buffer::v15_2_0::list::iterator_impl<t> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 22: (Migrator::decode_import_inode(CDentry*, ceph::buffer::v15_2_0::lis> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 23: (Migrator::decode_import_dir(ceph::buffer::v15_2_0::list::iterator_> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 24: (Migrator::handle_export_dir(boost::intrusive_ptr<MExportDir const>> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 25: (Migrator::dispatch(boost::intrusive_ptr<Message const> const&)+0x1> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 26: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 27: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, boo> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 28: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const>> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 29: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x10> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 30: (DispatchQueue::entry()+0x126a) [0x7fe2930a5aba] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 31: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe2931575d1] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 32: /lib64/libpthread.so.0(+0x81cf) [0x7fe291e451cf] Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: 33: clone() Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is neede> Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: --- begin dump of recent events --- Jan 19 04:49:40 naret-monitor03 ceph-63334166-d991-11eb-99de-40a6b72108d0-mds-cephfs-naret-monitor03-lqppte[4397]: terminate called recursively Jan 19 04:49:43 naret-monitor03 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)mds.cephfs.naret-monitor03.lqppte.service: Main process exited, code=exited, status=127/n/a Jan 19 04:49:43 naret-monitor03 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)mds.cephfs.naret-monitor03.lqppte.service: Failed with result 'exit-code'.

2 1

rbd_mirroring_delete_delay not removing images with snaps
by Tyler Brekke 24 Jan '23

24 Jan '23

We use the rbd-mirror as a way to migrate volumes between clusters. The process is enable mirroring on the image to migrate, demote on the primary cluster, promote on the secondary cluster, and then disable mirroring on the image. When we started using `rbd_mirroring_delete_delay` so we could retain a backup of the source image, we noticed volumes with unprotected snaps do not get purged from the trash. Previously, the image and all its snaps would be successfully removed after disabling mirroring. I would expect a similar function when using `rbd_mirroring_delete_delay` as well. Is rbd trash just overly cautious here? -- Tyler Brekke Senior Engineer I tbrekke(a)digitalocean.com ------------------------------ We're Hiring! <https://do.co/careers> | @digitalocean <https://twitter.com/digitalocean> | YouTube <https://www.youtube.com/digitalocean>

1 0

Pools and classes
by Massimo Sgaravatto 23 Jan '23

23 Jan '23

Dear all I have a ceph cluster where so far all OSDs have been rotational hdd disks (actually there are some SSDs, used only for block.db and wal.db) I now want to add some SSD disks to be used as OSD. My use case is: 1) for the existing pools keep using only hdd disks 2) create some new pools using only sdd disks Let's start with 1 (I didn't have added yet the ssd disks in the cluster) I have some replicated pools and some ec pools. The replicated pools are using a replicated_ruleset rule [*]. I created a new "replicated_hdd" rule [**] using the command: ceph osd crush rule create-replicated replicated_hdd default host hdd I then changed the crush rule of a existing pool (that was using 'replicated_ruleset') using the command: ceph osd pool set <poolname> crush_rule replicated_hdd This triggered the remapping of some pgs and therefore some data movement. Is this normal/expected, since for the time being I have only hdd osds ? Thanks, Massimo [*] rule replicated_ruleset { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } [**] rule replicated_hdd { id 7 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type host step emit }

2 3

trouble deploying custom config OSDs
by seccentral 23 Jan '23

23 Jan '23

Hi. I'm new to ceph, been toying around in a virtual environment (for now) trying to understand how to manage it. I made 3 vms in proxmox and provisioned a bunch of virtual drives to each. Bootstrapped following the quincy-branch official documentation. These are the drives: > /dev/sdb 128.00 GB sdb True False QEMU HARDDISK (HDD) > /dev/sdc 128.00 GB sdc True False QEMU HARDDISK (HDD) > /dev/sdd 32.00 GB sdd False False QEMU HARDDISK (SSD) This is the lvdisplay on /dev/sdd after creating two lvs: > db-0 dev0-db-0 -wi-a----- 16.00g > > db-1 dev0-db-0 -wi-a----- <16.00g My curiosity was to have OSDs with data=raw + block.db=lv created like this: > ceph-volume raw prepare --bluestore --data /dev/sdd --block.db /dev/mapper/dev0--db--0--db--0 This required tinkering with permissions and temporarily modifying /etc/ceph/ceph.keyring because by default it wasn't allowing access, RADOS complained about unauthorized client.boostrap-osd something but I got it to work eventually. (By the way, In a real environment, would RAW be of any benefit vs lvm everywhere ?) So now I have created 2 OSDs, each with the journal on the SSD and the data on the HDD. I repeated the steps on my other two boxes (btw, can't this be done from the local box via ceph cli ?) Now I am trying (and failing) to start OSD daemons on this host. I tried apply osd --all-available-devices, it tells me "Scheduled osd.all-available-devices update..." but nothing happens. I'm also not sure how to apply osds from a yaml file since that would provision them and .. they're already provisioned using the ceph-volume command above... right ? I'm having trouble getting a lot of things to work, this is just one of them and even if I feel nostalgic using mailing lists, It's inefficient. Is there any interactive community where I can find some people usually online and talk to them realtime like discord/slack etc ? I tried irc but most are afk. Thanks Sent with [Proton Mail](https://proton.me/) secure email.

2 3

Re: Ceph Disk Prediction module issues
by Nikhil Shah 23 Jan '23

23 Jan '23

Hey, did you ever find a resolution for this?

1 0

Set async+rdma in Ceph cluster
by Aristide Bekroundjo 21 Jan '23

21 Jan '23

Hi, I try to set rdma setting in cluster of 6 nodes (3 mon, 3 OSD Nodes and 10 OSDs on each OSD nodes). OS : CentOS Stream release 8. I've followed the step bellow, but I got an error. [root@mon1 ~]# cephadm shell Inferring fsid 9414e1bc-9061-11ed-90fc-00163e4f92ad Using recent ceph image quay.io/ceph/ceph@sha256:3cd25ee2e1589bf534c24493ab12e27caf634725b4449d50408fd5ad4796bbfa [ceph: root@mon1 /]# ceph config set global ms_type async+rdma 2023-01-21T11:11:49.182+0000 7fab5922e700 -1 Infiniband verify_prereq!!! WARNING !!! For RDMA to work properly user memlock (ulimit -l) must be big enough to allow large amount of registered memory. We recommend setting this parameter to infinity /usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = Worker*; _Alloc = std::allocator<Worker*>; std::vector<_Tp, _Alloc>::reference = Worker*&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed. Aborted (core dumped) [ceph: root@mon1 /]# With error I can see suggestion about ulimit, but with a containerized deployment, How can I configure properly async+rdma. Best regards,

1 0

Retrieve number of read/write operations for a particular file in Cephfs
by thanh son le 21 Jan '23

21 Jan '23

Hi, I have been studying the document from Ceph and Rados but I could not find any metrics to measure the number of read/write operations for each file. I understand that Cephfs is the front-end, the file is going to be stored as an object in the OSD and I have found that Ceph provides a Cache Tiering feature which also requires the monitor for read/write operation for each object. Could someone please give me guidance on how this is achieved? Thanks. Best regards, Le Thanh Son

2 1

RBD to fail fast/auto unmap in case of timeout
by Mathias Chapelain 20 Jan '23

20 Jan '23

Hello, We would like to run a RAID1 between a local storage and a RBD device. This would allow us to sustain network failures or Ceph failures and also give better read performance as we would set it up with write-mostly on RBD in mdadm. Basically we would like to implement https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme…. RAID1 is working well but if there is timeouts, the RBD volume won't fail and mdadm will not catch the broken device. Also the writes then hangs waiting for the network/RBD to come back. If we force unmap the RBD device then it fails as expected and writes can continue on other RAID1 device. We tried setting the `osd_request_timeout` to a small value (3 or 2 seconds) but it only gives us timeout in kernel logs: ``` libceph: tid 25792 on osd39 timeout rbd: rbd0: write at objno 602 0~512 result -110 rbd: rbd0: write result -110 print_req_error: 15 callbacks suppressed blk_update_request: timeout error, dev rbd0, sector 4931584 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 libceph: tid 25794 on osd39 timeout rbd: rbd0: write at objno 602 512~512 result -110 rbd: rbd0: write result -110 blk_update_request: timeout error, dev rbd0, sector 4931585 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0 ``` Is there something that we missed or is it currently impossible with kRBD to kind of "fail fast" in case of timeout and unmap/remove associated RBD devices? Or is there another client that can do what we want (ceph-nbd or with librbd)? We found this rook issue that is not really helpful but give insight https://github.com/rook/rook/issues/376. Thanks! -- Mathias Chapelain Storage Engineer Proton AG

1 0

ceph quincy rgw openstack howto
by Shashi Dahal 19 Jan '23

19 Jan '23

Hi, How to set up values for rgw_keystone_url and other related fields that are not possible to change via the GUI under cluster configuration. ? ceph qunicy is deployed using cephadm. -- Cheers, Shashi

2 1

journal fills ...
by Michael Lipp 19 Jan '23

19 Jan '23

I'm running Quincy and my journal fills with messages that I consider "debug" level such as: * ceph-mgr[1615]: [volumes INFO mgr_util] scanning for idle connections.. * ceph-mon[1617]: pgmap v1176995: 145 pgs: 145 active+clean; ... * ceph-mgr[1615]: [dashboard INFO request] ... * ceph-mgr[1615]: [pg_autoscaler INFO root] effective_target_ratio 0.0 0.0 0 1649254858752 I've never changed levels after installation and e.g. `ceph config show osd.0 | fgrep debug` does not show anything (and `ceph config show mgr.0`results in "Error ENOENT: no config state for daemon mgr.0". I have no ideas left, any hints? - Michael

1 0

journal fills ...
by Michael Lipp 19 Jan '23

19 Jan '23

I'm running Quincy and my journal fills with messages that I consider "debug" level such as: * ceph-mgr[1615]: [volumes INFO mgr_util] scanning for idle connections.. * ceph-mon[1617]: pgmap v1176995: 145 pgs: 145 active+clean; ... * ceph-mgr[1615]: [dashboard INFO request] ... * ceph-mgr[1615]: [pg_autoscaler INFO root] effective_target_ratio 0.0 0.0 0 1649254858752 I've never changed levels after installation and e.g. `ceph config show osd.0 | fgrep debug` does not show anything (and `ceph config show mgr.0`results in "Error ENOENT: no config state for daemon mgr.0". I have no ideas left, any hints? - Michael

1 0

Ceph rbd clients surrender exclusive lock in critical situation
by Frank Schilder 19 Jan '23

19 Jan '23

Hi all, we are observing a problem on a libvirt virtualisation cluster that might come from ceph rbd clients. Something went wrong during execution of a live-migration operation and as a result we have two instances of the same VM running on 2 different hosts, the source- and the destination host. What we observe now is the the exclusive lock of the RBD disk image moves between these two clients periodically (every few minutes the owner flips). We are pretty sure that no virsh commands possibly having that effect are executed during this time. The client connections are not lost and the OSD blacklist is empty. I don't understand why a ceph rbd client would surrender an exclusive lock in such a split brain situation, its exactly when it needs to hold on to it. As a result, the affected virtual drives are corrupted. The questions we have in this context are: Under what conditions does a ceph rbd client surrender an exclusive lock? Could this be a bug in the client or a ceph config error? Is this a known problem with libceph and libvirtd? Anyone else making the same observation and having some guidance? The VM hosts are on alma8 and we use the advanced virtualisation repo providing very recent versions of qemu and libvirtd. We have seen this floating exclusive lock before on mimic. Now we are on octopus and I can't really blame it on the old ceph version any more. We use opennebula as a KVM front-end. Thanks for any pointers! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

MDS crash in "inotablev == mds->inotable->get_version()"
by Kenny Van Alstyne 19 Jan '23

19 Jan '23

Hey all! I’ve run into an MDS crash on a cluster recently upgraded from Ceph 16.2.7 to 16.2.10. I’m hitting an assert nearly identical to this one gathered by the telemetry module: https://tracker.ceph.com/issues/54747 I have a new build compiling to test whether https://github.com/ceph/ceph/pull/43184/ makes a difference or not, when setting mds_inject_skip_replaying_inotable. Relevant logs are below, but I’m wondering if anyone has hit anything like this? Thanks in advance! === BEGIN LOG SNIPPET === -2> 2023-01-18T20:16:29.789+0000 7f6190243700 -1 log_channel(cluster) log [ERR] : journal replay alloc 0x10000000010 not in free [0x10000000011~0x3dc,0x100000003fb~0x1e8,0x100000005e5~0x2,0x100000009d4~0x2,0x1000005cc6d~0x4,0x10001c6b44e~0x4,0x10001cb91f4~0x1f4,0x10001cb93f4~0x3dd,0x10007582c15~0x279,0x10007582e90~0x1f4,0x10007583094~0xfff8a7cf6c] -1> 2023-01-18T20:16:29.789+0000 7f6190243700 -1 /builds/66321/e7c73776/ceph/-build//WORKDIR/ceph-16.2.10/src/mds/journal.cc: In function 'void EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)' thread 7f6190243700 time 2023-01-18T20:16:29.794189+0000 /WORKDIR/ceph-16.2.10/src/mds/journal.cc: 1577: FAILED ceph_assert(inotablev == mds->inotable->get_version()) ceph version 16.2.10 (e7c73776b3136f6d18a35febeb38f5fdd41be364) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14c) [0x7f619d548645] 2: /usr/lib/ceph/libceph-common.so.2(+0x27182f) [0x7f619d54882f] 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5815) [0x560bfd1c6935] 4: (EUpdate::replay(MDSRank*)+0x3c) [0x560bfd1c7ecc] 5: (MDLog::_replay_thread()+0xca9) [0x560bfd153de9] 6: (MDLog::ReplayThread::entry()+0xd) [0x560bfce78fdd] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f619cf29fa3] 8: clone() 0> 2023-01-18T20:16:29.793+0000 7f6190243700 -1 *** Caught signal (Aborted) ** in thread 7f6190243700 thread_name:md_log_replay ceph version 16.2.10 (e7c73776b3136f6d18a35febeb38f5fdd41be364) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f619cf34730] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19d) [0x7f619d548696] 5: /usr/lib/ceph/libceph-common.so.2(+0x27182f) [0x7f619d54882f] 6: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x5815) [0x560bfd1c6935] 7: (EUpdate::replay(MDSRank*)+0x3c) [0x560bfd1c7ecc] 8: (MDLog::_replay_thread()+0xca9) [0x560bfd153de9] 9: (MDLog::ReplayThread::entry()+0xd) [0x560bfce78fdd] 10: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f619cf29fa3] 11: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. === END LOG SNIPPET ===

1 0

Flapping OSDs on pacific 16.2.10
by J-P Methot 19 Jan '23

19 Jan '23

Hi, We have a full SSD production cluster running on Pacific 16.2.10 and deployed with cephadm that is experiencing OSD flapping issues. Essentially, random OSDs will get kicked out of the cluster and then automatically brought back in a few times a day. As an example, let's take the case of OSD.184 : -It flapped 9 times between January 15th and 17th with the following log message each time : 2023-01-15T16:33:19.903+0000 prepare_failure osd.184 from osd.49 is reporting failure:1 -On January 17th, it complains that there are slow ops and spam its logs with the following line : heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f346aa64700' had timed out after 15.000000954s The storage node itself has over 30 GB of ram still available in cache and the drives themselves only seldom peak at 100% usage and that never lasts more than a few seconds. CPU usage is also constantly around 5%. Considering there is no other error messages in any of the regular logs, including the systemd logs, why would this OSD not reply to heartbeats? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

4 6

ceph orch osd spec questions
by Wyll Ingersoll 18 Jan '23

18 Jan '23

Ceph Pacific 16.2.9 We have a storage server with multiple 1.7TB SSDs dedicated to the bluestore DB usage. The osd spec originally was misconfigured slightly and had set the "limit" parameter on the db_devices to 5 (there are 8 SSDs available) and did not specify a block_db_size. ceph layed out the original 40 OSDs and put 8 DBs across 5 of the SSDs (because of limit param). Ceph seems to have auto-sized the bluestore DB partitions to be about 45GB, which is far less than the recommended 1-4% (using 10TB drives). How does ceph-volume determine the size of the bluestore DB/WAL partitions when it is not specified in the spec? We updated the spec and specified a block_db_size of 300G and removed the "limit" value. Now we can see in the cephadm.log that the ceph-volume command being issued is using the correct list of SSD devices (all 8) as options to the lvm batch (--db-devices ...), but it keeps failing to create the new OSD because we are asking for 300G and it thinks there is only 44G available even though the last 3 SSDs in the list are empty (1.7T). So, it appears that somehow the orchestrator is ignoring the last 3 SSDs. I have verified that these SSDs are wiped clean, have no partitions or LVM, and no label (sgdisk -Z, wipefs -a). They appear as available in the inventory and not locked or otherwise in use. Also, the "db_slots" spec parameter is ignored in pacific due to a bug so there is no way to tell the orchestrator to use "block_db_slots". Adding it to the spec like "block_db_size" fails since it is not recognized. Any help figuring out why these SSDs are being ignored would be much appreciated. Our spec for this host looks like this: --- spec: data_devices: rotational: 1 size: '3TB:' db_devices: rotational: 0 size: ':2T' vendor: 'SEAGATE' block_db_size: 300G ---

1 1

Ceph-ansible: add a new HDD to an already provisioned WAL device
by Len Kimms 18 Jan '23

18 Jan '23

Hello all, we’ve set up a new Ceph cluster with a number of nodes which are all identically configured. There is one device vda which should act as WAL device for all other devices. Additionally, there are four other devices vdb, vdc, vdd, vde which use vda as WAL. The whole cluster was set up using ceph-ansible (branch stable-7.0) and Ceph version 17.2.0. Device configuration in osds.yml looks as follows: devices: [/dev/vdb, /dev/vdc, /dev/vdd, /dev/vde] bluestore_wal_devices: [/dev/vda] As expected vda contains four logical volumes for WAL each 1/4 of the overall vda disk size (‘ceph-ansible/group_vars/all.yml’ has default ‘block_db_size: -1’). After the initial setup, we’ve added an additional device vdf which should become a new OSD. The new OSD should use vda for WAL as well. This means the previous four WAL LVs have to be resized down to 1/5 and a new LV has to be added. Is it possible to retroactively add a new device to an already provisioned WAL device? We suspect that this is not possible because the ceph-bluestore-tool does not provide any way to shrink an existing BlueFS device. Only expanding is currently possible (https://docs.ceph.com/en/quincy/man/8/ceph-bluestore-tool/) Simply adding the new device to the devices list and rerunning the playbook does nothing. And so does only setting “devices: [/dev/vdf]” and “bluestore_wal_devices: [/dev/vda]”. In both cases vda is rejected because “Insufficient space (<10 extents) on vgs” which makes sense because vda is already fully used by the previous four OSD WALs. Thanks for the help and kind regards. Additional notes: - We’re testing pre-production on an emulated cluster hence the device names vdx and unusually small device sizes. - The output of `lsblk` after the initial setup looks as follows: ``` vda 252:0 0 8G 0 disk ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--3677c354--8d7d--4db9--a2b7--68aeb8248d40 253:2 0 2G 0 lvm ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--52d71122--b573--4077--9633--968c178612fd 253:4 0 2G 0 lvm ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--2d7eb467--cfb1--4a00--8a45--273932036599 253:6 0 2G 0 lvm └─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--d7b13b79--219c--4002--9e92--370dff7a5376 253:8 0 2G 0 lvm vdb 252:16 0 8G 0 disk └─ceph--49ddaa8b--5d8f--4267--85f9--5cac608ce53d-osd--block--861a53c7--ee57--4c5f--9546--1dd7cb0185ef 253:1 0 8G 0 lvm vdc 252:32 0 5G 0 disk └─ceph--1ed9ee91--e071--4ea4--9703--d56d84d9ae0a-osd--block--8aacb66a--e29b--4b7a--8ad5--a9fb1f81c6d6 253:3 0 5G 0 lvm vdd 252:48 0 5G 0 disk └─ceph--554cdd8b--e722--41a9--8f64--c09c857cc0dc-osd--block--4dee3e1b--b50d--4154--b2ff--80cadb67e2a0 253:5 0 5G 0 lvm vde 252:64 0 5G 0 disk └─ceph--5d58de32--ca55--4895--8ac7--af94ee07672e-osd--block--3f563f40--0c1e--4cca--9325--d9534cceb711 253:7 0 5G 0 lvm vdf 252:80 0 5G 0 disk ``` - Ceph status is happy and healthy: ``` cluster: id: ff043ce8-xxxx-xxxx-xxxx-e98d073c9d09 health: HEALTH_WARN mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum baloo-1,baloo-2,baloo-3 (age 13m) mgr: baloo-2(active, since 5m), standbys: baloo-3, baloo-1 mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 4m), 24 in (since 5m) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 7 pools, 177 pgs objects: 213 objects, 584 KiB usage: 98 MiB used, 138 GiB / 138 GiB avail pgs: 177 active+clean ```

2 2

[RFC] Detail view of OSD network I/O
by Nico Schottelius 18 Jan '23

18 Jan '23

Good morning ceph community, for quite some time I was wondering if it would not make sense to add an iftop alike interface to ceph that shows network traffic / iops on a per IP basis? I am aware of "rbd perf image iotop", however I am much more interested into a combined metric featuring 1) Which clients read/write to where and 2) inter OSDs traffic to see the total load on the cluster and being able to drill down. For example, the metric could look like this: -------------------------------------------------------------------------------- FROM TO Bytes/s Packets/s osd.0 [IP] -> [IP] osd.10 .. .. osd.0 [IP] -> [IP] client .. .. -------------------------------------------------------------------------------- Given that this table would be sortable by from/to/min-or-max bytes/min-or-max packets, this would allow spotting the And maybe a summarised view such as: -------------------------------------------------------------------------------- FROM IN Bytes/s OUT Bytes/s IN Packets/s OUT Packets/s osd.0 [IP] osd.10 [IP] -------------------------------------------------------------------------------- This way it would be nicely possible to identify high load. If it was combined with average/current latency, it would potentially also be able to find the bottlenecks in the cluster. From my perspective, easily combining client + intra cluster traffic would be very helpful. What do you think, does that make sense, does it already exist or how do you approach this? Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Ceph Community Infrastructure Outage
by Mike Perez 18 Jan '23

18 Jan '23

Hi everyone, From November into January, we experienced a series of outages with the Ceph Community Infrastructure and its services: - Mailing lists - https://lists.ceph.io - Sepia (testing infrastructure) - https://wiki.sepia.ceph.com - https://pulpito.ceph.com - https://chacra.ceph.com - https://shaman.ceph.com - VPN to access testing services - Etherpad - https://pad.ceph.com - Images: - https://quay.ceph.io - Git mirror - https://git.ceph.com - https://ceph.io - Telemetry <https://telemetry-public.ceph.com/> These services are now mostly restored, but we did experience some data loss, notably in our mailing lists. We have restored them from backups, but subscription changes after July 2021 need to be repeated. If you subscribed or unsubscribed since then, please check your settings with the appropriate list at https://lists.ceph.io. If your posts to our mailing lists are now needing approval, that is also an indication that you need to re-subscribe to the appropriate lists. Keep an eye out for emails with subject lines such as “Your message to ceph-users(a)ceph.io awaits moderator approval”. When the community infrastructure was first created in late 2014, the VM cluster management software selected by the team came with the benefit of being widely entrenched and familiar to the lab administrators but didn't support Ceph as a storage backend at the time. As services grew, we relied more and more on its legacy storage solution, which was never migrated to Ceph. Over the last few months, this legacy storage solution had several instances of silent data corruption, rendering the VMs unbootable, taking down various services, and requiring restoration from backups in many cases. We are moving these services to a more reliable, mostly container-based, infrastructure backed by Ceph, and planning for longer-term improvements to monitoring, backups, deployment, and other pieces of the project infrastructure. This event highlights the need to better support the infrastructure. A handful of contributors have stepped up to restore these services, but we need an invested team focused. If you or your company is looking for a great way to contribute to the Ceph community, this could be your opportunity. Please contact council(a)ceph.io if you can provide time to contribute to the Ceph Community Infrastructure and would like to join the team. You can also join the upstream #sepia slack channel to participate in these discussions using this link: https://join.slack.com/t/ceph-storage/shared_invite/zt-1n1eh6po5-PF9sokUSoo… Unfortunately, these events have slowed down our upstream development and releases. We are currently working on publishing the next Pacific point release. The development freeze and release deadline for the Reef release will likely be pushed out, and more discussions to follow in the Ceph Leadership Team meetings. - The Ceph Leadership Team

2 1

Stable erasure coding CRUSH rule for multiple hosts?
by aschmitz 18 Jan '23

18 Jan '23

Hi folks, I have a small cluster of three Ceph hosts running on Pacific. I'm trying to balance resilience and disk usage, so I've set up a k=4 m=2 pool for some bulk storage on HDD devices. With the correct placement of PGs this should allow me to take any one host offline for maintenance. I've written this CRUSH rule for that purpose: rule erasure_k4_m2_hdd_rule { id 3 type erasure step take default class hdd step choose indep 3 type host step chooseleaf indep 2 type osd step emit } This should pick three hosts, and then two OSDs from each, which at least ensures that no host has more than two OSDs. This appears to work correctly, but I'm running into an odd situation when adding additional OSDs to the cluster: sometimes the hosts flip order in a PG's set, resulting in unnecessary remapping work. For example, I have one PG that changed from OSDs [0,13,7,9,3,5] to [0,13,3,5,7,9]. (Note that the middle two and last two sets of OSDs have swapped with one another.) From a quick perusal of other PGs that are being moved, the two OSDs within a host never appear to be rearranged, but the set of hosts that are chosen may be shuffled. Is there something I'm missing that would make this rule more stable in the face of OSD addition? (I'm wondering if the host choosing component should be "firstn" rather than "indep", even though the discussion at https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/#crushmapr… implies indep is preferable in EC pools.) I don't have current plans to expand beyond a three-host cluster, but if there's an alternative way to express "not more than two OSDs per host", that could be helpful as well. Any insights or suggestions would be appreciated. Thanks, aschmitz

2 1

bidirectional rbd-mirroring
by Aielli, Elia 18 Jan '23

18 Jan '23

Hi all, I've a working couple of cluster configured with rbd mirror, Master cluster is production, Backup cluster is DR. Right now all is working good with Master configured in "tx-only" and Backup in "rx-tx". I'd like to modify Master direction to rx-tx so I'm already prepared for a failover after a disaster has happened, but while I'm doin so, I face this error and i'm stuck: ceph version 15.2.17 (694d03a6f6c6e9f814446223549caf9a9f60dba0) octopus (stable) Ceph user able to operate on Master is rbd-mirror.master, while on Backup is rbd-mirror.backup On Master cluster I've my ceph.conf and backup.conf, and on Backup cluster I've ceph.conf and master.conf Keyrings has been copied correctly. I've change direction without any problem, but when i try to configure the peer with this command, i receive following error: root@master# rbd mirror pool peer add <my_pool> client.rbd-mirror.backup@backup rbd: multiple RX peers are not currently supported And when I check my pool info, i have the "Client:" section empty (while the one on my DR is populated with client.rbd-mirror.master" Can someone lend me a hand? Is this something I can't do or simply I'm using the wrong commands? Thanks in advance! Elia

2 2

Ceph User + Dev Monthly January Meetup
by Neha Ojha 18 Jan '23

18 Jan '23

Hi everyone, This month's Ceph User + Dev Monthly meetup is on January 19, 15:00-16:00 UTC. There are some topics in the agenda regarding RGW backports, please feel free to add other topics to https://pad.ceph.com/p/ceph-user-dev-monthly-minutes. Hope to see you there! Thanks, Neha

1 0

Dashboard access to CephFS snapshots
by Robert Sander 18 Jan '23

18 Jan '23

Hi, The dashboard has a simple CephFS browser where we can set quota and snapshots for the directories. When a directory has the "other" permission bits unset, i.e. only access for user and group, the dashboard displays an error: Failed to execute CephFS opendir failed at /path/to/dir/.snap: Permission denied [Errno 13] It can be reproduced in Ceph 17.2.5 by creating the directory and using "chmod o= /path/to/dir" to not allow "other". How does the dashboard access the contents of the CephFS? It looks like the MGR uses something like the nobody account. Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

Re: opnesuse rpm repos
by Eugen Block 17 Jan '23

17 Jan '23

Hi, the last RPMs I'm aware of for openSUSE are for Pacific, and we get them from software.opensuse.org. We used Pacific for the switch to cephadm and containers in our openSUSE based ceph cluster. I don't see any opensuse directory underneath download.ceph.com, were they actually present in the past? I've never used that repo. But the general development is directed to containers. Regards, Eugen Zitat von Mazzystr <mazzystr(a)gmail.com>: > Hello, > I ran the following on OpenSuse > > cephadm add-repo --release quincy > > > The command landed /etc/zypp/repos.d/ceph.repo and contents > > [Ceph] > name=Ceph $basearch > baseurl=https://download.ceph.com/rpm-quincy/opensuse/$basearch > enabled=1 > gpgcheck=1 > gpgkey=https://download.ceph.com/keys/release.gpg > > [Ceph-noarch] > name=Ceph noarch > baseurl=https://download.ceph.com/rpm-quincy/opensuse/noarch > enabled=1 > gpgcheck=1 > gpgkey=https://download.ceph.com/keys/release.gpg > > [Ceph-source] > name=Ceph SRPMS > baseurl=https://download.ceph.com/rpm-quincy/opensuse/SRPMS > enabled=1 > gpgcheck=1 > gpgkey=https://download.ceph.com/keys/release.gpg > > Apparently these repos don't exist. > > Do we plan on building for OpenSuse or isd the plan to migrate to > containers? > > Thanks, > /Chris > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Mysterious HDD-Space Eating Issue
by matthew＠peregrineit.net 17 Jan '23

17 Jan '23

Hi Guys, I've got a funny one I'm hoping someone can point me in the right direction with: We've got three identical(?) Ceph nodes running 4 OSDs, Mon, Man, and iSCSI G/W each (we're only a small shop) on Rocky Linux 8 / Ceph Quincy. Everything is running fine, no bottle-necks (as far as we can see) and the Cluster is holding up very well. However, one of the boxes is constantly running out of space on the /var mount. Its 16 GiB in size, and it only takes a day or three to fill up, thus taking it's monitor service out of quorum. The thing is, I can't find *what's* taking up all the space. At first we thought it was an overly large log file, but I've done searches to find the largest files, etc, and nothing is showing up (that I can find) - ie the log files on this box are comparable with the log files on the other two boxes and the other two boxes are sitting at around 10% full (via a df-H), while the problem box is at around 85% and growing (at time of posting). Another interesting point is that the problem box, unrelated to this issue, was rebooted recently and when it came back on-line the space-issue was gone ie the /var mount was back down to around the 10% mark. This suggests to me its some sort of "temporary" journal/log/dump/whatever/? that was "reset" (cleaned-up?) via the reboot. I've had a look at the logs but I'm not sure what I should be looking for - so I don't even know if I'm looking in the *correct* logs... Anyone got any ideas? I mean, rebooting the server every couple of days is not really a practical solution, and neither is turning off the monitor service on the box, and increasing the size of the /var mount just seems like it'll postpone the issue. Any help would be greatly appreciated. Cheers Dulux-Oz

4 4

._handle_peer_banner peer [v2:***,v1:***] is using msgr V1 protocol
by Frank Schilder 17 Jan '23

17 Jan '23

Hi all, on an octopus latest cluster I see a lot of these log messages: Jan 13 20:00:25 ceph-21 journal: 2023-01-13T20:00:25.366+0100 7f47702b8700 -1 --2- [v2:192.168.16.96:6826/5724,v1:192.168.16.96:6827/5724] >> [v2:192.168.16.93:6928/3503064,v1:192.168.16.93:6929/3503064] conn(0x55c867624400 0x55c7e9dfa800 unknown :-1 s=BANNER_CONNECTING pgs=22826 cs=73364 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner peer [v2:192.168.16.93:6928/3503064,v1:192.168.16.93:6929/3503064] is using msgr V1 protocol These addresses are on the replication network and both hosts are OSD hosts. What is the reason for these messages and how can I fix it? Thanks a lot! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 3

PG_BACKFILL_FULL
by Iztok Gregori 17 Jan '23

17 Jan '23

Hi to all! We are in a situation where we have 3 PG in "active+remapped+backfill_toofull". It happened when we executed a "gentle-reweight" to zero of one OSD (osd.77) to swap it with a new one (the current one registered some read errors and it's to be replaced just-in-case). > # ceph health detail: > [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull > pg 10.46c is active+remapped+backfill_toofull, acting [77,607,96] > pg 10.8ad is active+remapped+backfill_toofull, acting [577,152,77] > pg 10.b15 is active+remapped+backfill_toofull, acting [483,348,77] Our cluster is a little unbalanced and we have 7 OSD nearfull (I think it's because we have 4 nodes with 6 TB disks and the other 19 have 10 TB, but should be unrelated, why is the cluster unbalanced I mean, to the backfill_toofull ) not by too much (less then 88%). I'm not too much worried about it, we will add new storage this month (if the servers will arrive) and we will get rid of the old 6 TB servers. If I dump the PGs I see, if I'm not mistaken, that the osd.77 will be "replaced" by the osd.60, which is one of the nearfull ones (the top one with 87.53% used). > # ceph pg dump: > > 10.b15 37236 0 0 37236 0 155249620992 0 0 5265 5265 active+remapped+backfill_toofull 2023-01-16T14:45:46.155801+0100 305742'144106 305742:901513 [483,348,60] 483 [483,348,77] 483 305211'144056 2023-01-11T10:20:56.600135+0100 305211'144056 2023-01-11T10:20:56.600135+0100 0 > 10.8ad 37518 0 0 37518 0 156345024512 0 0 5517 5517 active+remapped+backfill_toofull 2023-01-16T14:45:38.510038+0100 305213'142117 305742:937228 [577,60,152] 577 [577,152,77] 577 303828'142043 2023-01-06T17:52:02.523104+0100 303334'141645 2022-12-20T17:39:22.668083+0100 0 > 10.46c 36710 0 0 36710 0 153023443456 0 0 8172 8172 active+remapped+backfill_toofull 2023-01-16T14:45:29.284223+0100 305298'141437 305741:877331 [60,607,96] 60 [77,607,96] 77 304802'141358 2023-01-08T21:39:23.469198+0100 304363'141349 2023-01-01T18:13:45.645494+0100 0 > # ceph osd df: > 60 hdd 5.45999 1.00000 5.5 TiB 4.8 TiB 697 GiB 128 MiB 0 B 697 GiB 87.53 1.29 37 up In this situation was the correct way to address the problem? reweight-by-utilization the osd.60 to free up space (the OSD is a 6 TB disk, and other OSD on the same host are nearfull)? There is other way to manually map a PG to a different OSD? Thank you for your attention Iztok Gregori

3 4

Filesystem is degraded, offline, mds daemon damaged
by bpurvis＠kfpl.ca 17 Jan '23

17 Jan '23

I am really hoping you can help. THANKS in advance. I have inherited a Docker swarm running CEPH but I know very little about it. Current I have an unhealthy ceph environment that will not mount my data drive. Its a cluster of 4 vm servers. docker01,docker02, docker03, docker-cloud CL has the /data that is on a separate drive, currently failing to mount. how can I recover this without loosing the data? on server docker-cloud, mount /data returns: mount error 113 = No route to host docker ps is healthy on all nodes. bc81d14dde92 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-mds d4fecec5e0e8 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-osd 482ba41803af ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 34 minutes ceph-mgr d6a5c44179c7 ceph/daemon:latest-mimic "/opt/ceph-container…" 2 years ago Up 32 minutes ceph-mon ceph -s: cluster: id: 7a5b2243-8e92-4e03-aee7-aa64cea666ec health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged noout,noscrub,nodeep-scrub flag(s) set clock skew detected on mon.docker02, mon.docker03, mon.docker-cloud mons docker-cloud,docker01,docker02,docker03 are low on available space services: mon: 4 daemons, quorum docker01,docker02,docker03,docker-cloud mgr: docker01(active), standbys: docker02, docker03, docker-cloud mds: cephfs-0/1/1 up , 4 up:standby, 1 damaged osd: 4 osds: 4 up, 4 in flags noout,noscrub,nodeep-scrub data: pools: 2 pools, 256 pgs objects: 194.2 k objects, 241 GiB usage: 499 GiB used, 1.5 TiB / 2.0 TiB avail pgs: 256 active+clean

2 1

ceph orch cannot refresh
by Nicola Mori 17 Jan '23

17 Jan '23

Dear Ceph users, after a host failure in my cluster (quincy 17.2.3 managed by cephadm) it seems that ceph orch got somehow stuck and it cannot operate. For example, it seems that it cannot refresh the status of several services since about 20 hours: # ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 3m ago 3M count:1 crash 9/10 20h ago 3M * grafana ?:3000 1/1 3m ago 3M count:1 mds.wizard_fs 0/3 <deleting> 13h bofur;balin;aka;count:3 mds.wizardfs 2/3 20h ago 70m bofur;balin;aka;count:3 mgr 2/2 20h ago 15m bofur;balin;count:2 mon 4/5 20h ago 93m bofur;balin;aka;romolo;dwalin;count:5 node-exporter ?:9100 9/10 20h ago 3M * osd 24 3m ago - <unmanaged> osd.all-available-devices 72 20h ago 4w * prometheus ?:9095 1/1 3m ago 3M count:1 The failed machine (named bifur) is offline but still in the cluster since I'm planning to restore it: # ceph orch host ls HOST ADDR LABELS STATUS aka 172.16.253.7 _admin balin 172.16.253.3 bifur 172.16.253.5 _admin Offline bofur 172.16.253.2 _admin dwalin 172.16.253.10 ogion 172.16.253.6 _no_autotune_memory prestno 172.16.253.9 remolo 172.16.253.1 rokanan 172.16.253.8 romolo 172.16.253.4 10 hosts in cluster Since this machine hosted a mon I tried to redeploy it with: # ceph orch apply mon --placement="5 bofur balin aka romolo dwalin" but even if ceph orch ls shows that the mons should currently be on the machines specified buy --placement (see above) it seems that somehow the mon on bifur is somehow still present in ceph orch status, e.g. # ceph orch restart mon Scheduled to restart mon.aka on host 'aka' Scheduled to restart mon.balin on host 'balin' Scheduled to restart mon.bifur on host 'bifur' Scheduled to restart mon.bofur on host 'bofur' Scheduled to restart mon.romolo on host 'romolo' I manually restarted all the mon and mgr daemons on online hosts to no avail. At this point I am clueless, so any help is greatly appreciated. Nicola

2 1

OSD slow ops warning not clearing after OSD down
by Frank Schilder 17 Jan '23

17 Jan '23

Dear cephers, I have a strange problem. An OSD went down and recovery finished. For some reason, I have a slow ops warning for the failed OSD stuck in the system: health: HEALTH_WARN 430 slow ops, oldest one blocked for 36 sec, osd.580 has slow ops The OSD is auto-out: | 580 | ceph-22 | 0 | 0 | 0 | 0 | 0 | 0 | autoout,exists | It is probably a warning dating back to just before the fail. How can I clear it? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 7

Unable to subscribe
by Abhinav Singh 17 Jan '23

17 Jan '23

I m unable to subscribe even after sending mail to ceph-leave-user email id. Is there any other way to unsbuscribe? Thanks & Regards Abhinav Singh

1 0

Re: large omap objects in the .rgw.log pool
by Ramin Najjarbashi 17 Jan '23

17 Jan '23

I am experiencing the same problem with 'Large omap object' in Ceph, so I wrote a script to find the large objects in the pool, count the number of "omap" in each object, and compare it with the value set for large_omap_object_key_threshold in the Ceph configuration. https://gist.github.com/RaminNietzsche/0297e9163834c050234686f5b4acb1a4 On Sun, Oct 30, 2022 at 5:12 AM Sarah Coxon <sazzle2611(a)gmail.com> wrote: > Hi Anthony, > > Thank you for getting back to me, the post you sent was helpful although > the log in that post was usage log and there aren't the same options for > the sync error log. > > I understand that I can get rid of the warning by increasing the > threshold but I would really like to get rid of the old data if possible. > > The sync error log is full of errors from buckets that have since been > deleted and the sync errors are over 2 years old. Trimming the sync error > log does nothing. > > From everything I've read this is an issue with having a multisite > configuration and removing buckets doesn't clear previous data properly > https://www.spinics.net/lists/ceph-devel/msg45359.html > > I already knew this as in I have been manually deleting metadata and index > data for buckets a while after clearing them of objects and deleting the > bucket itself (but making note of the bucket id to use in the later > commands) > > I have found this command to get object ids in the sync error log related > to the deleted buckets > > radosgw-admin sync error list | jq '.[0].entries[] | > select(.name|test("^company-report-images.")) | .id' > > but I don't know how to get rid of them and I really don't want to screw up > the whole setup by deleting the wrong thing. > > On Thu, Oct 27, 2022 at 9:52 AM Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > > > This prior post > > > > > > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2QNKWK642LW… > > > > may help. You can bump up the warning threshold to make the warning go > > away - a few releases ago it was reduced to 1/10 of the prior value. > > > > There’s also information about trimming usage logs and for removing > > specific usage log objects. > > > > > On Oct 27, 2022, at 4:05 AM, Sarah Coxon <sazzle2611(a)gmail.com> wrote: > > > > > > Hey, I would really appreciate any help I can get on this as googling > has > > > led me to a dead end. > > > > > > We have 2 data centers each with 4 servers running ceph on kubernetes > in > > > multisite config, everything is working great but recently the master > > > cluster changed status to HEALTH_WARN and the issues are large omap > > objects > > > in the .rgw.log pool. Second cluster is still HEALTH_OK > > > > > > Viewing the sync error log from master shows a lot of very ancient logs > > > related to a bucket that has since been deleted. > > > > > > Is there any way to clear this log? > > > > > > bash-4.4$ radosgw-admin sync error list | wc -l > > > 352162 > > > > > > I believe, although I'm not sure that this is a massive part of the > data > > > stored in the .rgw.log pool, I haven't been able to find any info on > this > > > except for several other posts about clearing the error log but none of > > > them had a resolution. > > > > > > I am tempted to increase the PG's for this pool from 16 to 32 to see if > > it > > > helps but holding off because that is not an ideal solution just to get > > rid > > > of this warning when all I want is to get rid of the errors related to > a > > > bucket that no longer exists. > > > > > > Thanks to anyone that can offer advice! > > > > > > Sarah > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

pg mapping verification
by Christopher Durham 17 Jan '23

17 Jan '23

Hi, For a given crush rule and pool that uses it, how can I verify hat the pgs in that pool folllow the rule? I have a requirement to 'prove' that the pgs are mapping correctly. I see: https://pypi.org/project/crush/ This allows me to read in a crushmap file that I could then use to verify a pg with some scripting, but this pypi is very old and seems not to be maintained or updatedsince 2017. I am sure there is a way, using osdmaptool or something else, but it is not obvious. Before i spend alot of time searching, I thought I would ask here. Basically, having a list of pgs like this: [[1,2,3,4,5],[2,3,4,5,6],...] Given a read-in crushmap and a specific rule therein, I want to verify that all pgs in my list are consistent with the rule specified. Let me know if there is a proper way to do this, and thanks. -Chris

3 4

osd_memory_target values
by Mevludin Blazevic 17 Jan '23

17 Jan '23

Hi all, for a ceph cluster with RAM size of 256 GB per node, I would increase the osd_memory_target from default 4GB up to 12GB. Through the ceph dashboard, different values are given to set the new value (global, mon, ..., osd). Is there any difference between them? From my point of view, I should simply set the osd value to 12 GB, am I right? Best, Mevludin

2 1

Building Ceph containers
by Matthew Vernon 17 Jan '23

17 Jan '23

Hi, Is it possible/supported to build Ceph containers on Debian? The build instructions[0] talk about building packages (incl. .debs), but now building containers. Cephadm only supports containerised deployments, but our local policy is that we should only deploy containers we've built ourselves. Is it still the case that only building centos-based images is supported? Perhaps with a follow-up question of why container building isn't supported on the range of platforms that package building is supported on if containerised deployments are where people are meant to be going... Thanks, Matthew [0] https://docs.ceph.com/en/quincy/install/build-ceph/

1 0

Useful MDS configuration for heavily used Cephfs
by E Taka 16 Jan '23

16 Jan '23

Ceph 17.2.5: Hi, I'm looking for a reasonable and useful MDS configuration for a – in future, no experiences until now – heavily used CephFS (~100TB). For example, does it make a difference to increase the mds_cache_memory_limit or the number of MDS instances? The hardware does not set any limits, I just want to know where the default values can be optimized usefully before problem occur. Thanks,

4 5

RGW - large omaps even when buckets are sharded
by Boris Behrens 16 Jan '23

16 Jan '23

Hi, since last week the scrubbing results in large omap warning. After some digging I've got these results: # searching for indexes with large omaps: $ for i in `rados -p eu-central-1.rgw.buckets.index ls`; do rados -p eu-central-1.rgw.buckets.index listomapkeys $i | wc -l | tr -d '\n' >> omapkeys echo " - ${i}" >> omapkeys done $ sort -n omapkeys | tail -n 15 212010 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.0 212460 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.3 212466 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.10 213165 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2342226177.1.4 354692 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.7 354760 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.5 354799 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.1 355040 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.10 355874 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.2 355930 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.3 356499 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.6 356583 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.8 356925 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.4 356935 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.9 358986 - .dir.ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1.0 So I have a bucket (ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1) with 11 shards where each shard got around 350k omapkeys. When checking what bucket it is is get a total different number: $ radosgw-admin bucket stats --bucket-id=ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1 { "bucket": "bucket", "num_shards": 11, "tenant": "", "zonegroup": "da651dc1-2663-4e1b-af2e-ac4454f24c9d", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2421332952.1", "marker": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2296333939.13", "index_type": "Normal", "owner": "user", "ver": "0#45265,1#44764,2#44631,3#44777,4#44859,5#44637,6#44814,7#44506,8#44853,9#44764,10#44813", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0", "mtime": "2022-11-16T08:34:17.298979Z", "creation_time": "2021-11-16T09:13:34.480637Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#", "usage": { "rgw.main": { "size": 66897607205, "size_actual": 68261179392, "size_utilized": 66897607205, "size_kb": 65329695, "size_kb_actual": 66661308, "size_kb_utilized": 65329695, "num_objects": 663369 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } It got 11 shards, with a total of 663k files. radosgw-admin bucket limit check gives 60k objects per shard. After getting a list with all omapkeys (total of 3917043) I see that entries look like this (only in less, not in cat) - the ^@ char: $ grep -aF object1 2421332952.1_omapkeys object1 object1^@v910^@i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK object1^@v913^@iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ <80>1000_object1^@i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK <80>1000_object1^(a)iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ <80>1001_object1 I also pulled the whole bucket index of said bucket via radosgw-admin bi list --bucket bucket > bucket_index_list and searched via jq for the object1: $ jq '.[] | select(.entry.name == "object1")' bucket_index_list { "type": "plain", "idx": "object1", "entry": { "name": "object1", "instance": "", "ver": { "pool": -1, "epoch": 0 }, "locator": "", "exists": "false", "meta": { "category": 0, "size": 0, "mtime": "0.000000", "etag": "", "storage_class": "", "owner": "", "owner_display_name": "", "content_type": "", "accounted_size": 0, "user_data": "", "appendable": "false" }, "tag": "", "flags": 8, "pending_map": [], "versioned_epoch": 0 } } { "type": "plain", "idx": "object1\u0000v910\u0000i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "entry": { "name": "object1", "instance": "3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "ver": { "pool": -1, "epoch": 0 }, "locator": "", "exists": "false", "meta": { "category": 0, "size": 0, "mtime": "2022-12-16T00:00:28.651053Z", "etag": "", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "", "accounted_size": 0, "user_data": "", "appendable": "false" }, "tag": "delete-marker", "flags": 7, "pending_map": [], "versioned_epoch": 5 } } { "type": "plain", "idx": "object1\u0000v913\u0000iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "entry": { "name": "object1", "instance": "PVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "ver": { "pool": 11, "epoch": 2375707 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 10858, "mtime": "2021-12-15T12:05:30.351753Z", "etag": "8cc8ba9599322c17af56996bc0a85af0", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "image/jpeg", "accounted_size": 10858, "user_data": "", "appendable": "false" }, "tag": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644265.64076989", "flags": 1, "pending_map": [], "versioned_epoch": 2 } } { "type": "instance", "idx": "�1000_object1\u0000i3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "entry": { "name": "object1", "instance": "3nb5Cdp00wrt3Phhbn4MgwTcsM7sdwK", "ver": { "pool": -1, "epoch": 0 }, "locator": "", "exists": "false", "meta": { "category": 0, "size": 0, "mtime": "2022-12-16T00:00:28.651053Z", "etag": "", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "", "accounted_size": 0, "user_data": "", "appendable": "false" }, "tag": "delete-marker", "flags": 7, "pending_map": [], "versioned_epoch": 5 } } { "type": "instance", "idx": "�1000_object1\u0000iPVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "entry": { "name": "object1", "instance": "PVPdb60UlfOu4Mwzr.oqojwWzRdgheZ", "ver": { "pool": 11, "epoch": 2375707 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 10858, "mtime": "2021-12-15T12:05:30.351753Z", "etag": "8cc8ba9599322c17af56996bc0a85af0", "storage_class": "", "owner": "user", "owner_display_name": "user", "content_type": "image/jpeg", "accounted_size": 10858, "user_data": "", "appendable": "false" }, "tag": "ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644265.64076989", "flags": 1, "pending_map": [], "versioned_epoch": 2 } } Does anyone know what is happening here? And what should I do about the large omap objects? Reshard again?

1 0

User access
by Rhys Powell 14 Jan '23

14 Jan '23

rhys.g.powell(a)gmail.com

1 0

Remove failed multi-part uploads?
by rhys.g.powell＠gmail.com 14 Jan '23

14 Jan '23

Hello, We are running an older version of ceph - 14.2.22 nautilus We have a radosgw/s3 implementation and had some issues with multi-part uploads failing to complete. We used s3cmd to delete the failed uploads and clean out the bucket, but when reviewing the space utilization of buckets, it seems this one is still consuming space: [ ~]# radosgw-admin bucket stats --bucket=BUCKETNAME { "bucket": "BUCKETNAME", "num_shards": 32, "tenant": "", "zonegroup": "c73e02d6-d479-4cdc-bf86-8b09f0a9f6ba", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "50ee73bc-bc08-4f9f-9d5b-4492cb4c5e77.1689003.1695", "marker": "50ee73bc-bc08-4f9f-9d5b-4492cb4c5e77.1689003.1695", "index_type": "Normal", "owner": "BUCKETNAME", "ver": "0#47066,1#30480,2#42797,3#36437,4#47308,5#33285,6#37127,7#24292,8#44567,9#34273,10#29402,11#36228,12#48153,13#32665,14#42314,15#21143,16#34319,17#42818,18#39301,19#23897,20#26225,21#50957,22#39706,23#29723,24#49619,25#44974,26#44020,27#22505,28#46702,29#49390,30#27263,31#21515", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0", "mtime": "2021-02-08 13:06:13.311932Z", "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551613 }, "rgw.main": { "size": 34247260247640, "size_actual": 34247284682752, "size_utilized": 34247260247640, "size_kb": 33444590086, "size_kb_actual": 33444613948, "size_kb_utilized": 33444590086, "num_objects": 340627 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 0 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } I see under the usage.rgw.main.size_kb_actual the value is 33444613948, or roughly 30TB When I use the radosgw-admin tool to list objects, I can see many failed multi-part uploads: [ ~]# radosgw-admin bucket list --bucket BUCKETNAME | jq '.[] | "\(.name), \(.meta.mtime), \(.meta.size)"' "_multipart_chi-pl-clh-shard-0-0-0-2021-02-10.tar.gz.2~07YXhKKZn2XYy-6F0itVB4tpuBm1q1J.1, 2021-02-10 00:57:08.033082Z, 4194304" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-10.tar.gz.2~07YXhKKZn2XYy-6F0itVB4tpuBm1q1J.2, 2021-02-10 00:56:36.463099Z, 8794011" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-10.tar.gz.2~b6-C6I3rky3V2Wh4H56jhsfVjvvTMj2.1, 2021-02-10 00:38:44.572199Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-10.tar.gz.2~b6-C6I3rky3V2Wh4H56jhsfVjvvTMj2.2, 2021-02-10 00:38:48.680330Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-10.tar.gz.2~b6-C6I3rky3V2Wh4H56jhsfVjvvTMj2.3, 2021-02-10 00:38:52.232674Z, 95445231" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~R8SwLZMVNM5kL4Ov7sX47mXdEJf0hfu.1, 2021-02-11 00:30:55.489965Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~R8SwLZMVNM5kL4Ov7sX47mXdEJf0hfu.2, 2021-02-11 00:30:58.832752Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~R8SwLZMVNM5kL4Ov7sX47mXdEJf0hfu.3, 2021-02-11 00:31:01.188868Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~R8SwLZMVNM5kL4Ov7sX47mXdEJf0hfu.4, 2021-02-11 00:30:53.035172Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~R8SwLZMVNM5kL4Ov7sX47mXdEJf0hfu.5, 2021-02-11 00:30:21.359861Z, 12448760" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~mPN97GOqO8E93gqVUbt_esJfB4kLu2h.1, 2021-02-11 00:11:52.163319Z, 4194304" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~mPN97GOqO8E93gqVUbt_esJfB4kLu2h.2, 2021-02-11 00:11:48.293292Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~mPN97GOqO8E93gqVUbt_esJfB4kLu2h.3, 2021-02-11 00:11:55.320413Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~mPN97GOqO8E93gqVUbt_esJfB4kLu2h.4, 2021-02-11 00:11:55.039628Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-11.tar.gz.2~mPN97GOqO8E93gqVUbt_esJfB4kLu2h.5, 2021-02-11 00:11:26.493213Z, 2005541" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-12.tar.gz.2~05JmbiZqt8tvgVmJ3Ef6WEzBa3Jla7L.1, 2021-02-12 00:53:24.453273Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-12.tar.gz.2~05JmbiZqt8tvgVmJ3Ef6WEzBa3Jla7L.2, 2021-02-12 00:54:00.743677Z, 9835956" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-12.tar.gz.2~90wJZ6jaWa6BaQC88e9YdXJwsqyme3u.1, 2021-02-12 00:59:24.943370Z, 104857600" "_multipart_chi-pl-clh-shard-0-0-0-2021-02-12.tar.gz.2~90wJZ6jaWa6BaQC88e9YdXJwsqyme3u.10, 2021-02-12 00:56:56.621609Z, 4194304" ... However, when I try to delete one of these object via radosgw-admin, I receive an error that the object is not found: [ ~]# radosgw-admin object rm --bucket BUCKETNAME --object=_multipart_chi-pl-clh-shard-0-0-0-2021-04-17.tar.gz.2~CVL_xbfGjdDckHe_hpJxoUSynjotOtR.18 ERROR: object remove returned: (2) No such file or directory When I list object via S3 API, none are found: [ minio-binaries]# ./mc ls BUCKETNAME [2021-02-08 08:06:13 EST] 0B BUCKETNAME/ [ minio-binaries]# ./mc ls BUCKETNAME/FOLDER [ minio-binaries]# ./mc ls BUCKETNAME/FOLDER [ minio-binaries]# ./mc ls --incomplete BUCKETNAME/FOLDER You have mail in /var/spool/mail/root [ minio-binaries]# I am wondering if I were to delete the bucket BUCKETNAME with the --purge option, would that remove the objects? Are the objects actually there? How can I confirm the objects exist? If I delete the bucket, how can I confirm the objects are gone? Any help would be greatly appreciated! Rhys

1 0

MDS error
by André de Freitas Smaira 14 Jan '23

14 Jan '23

Hello! Yesterday we found some errors in our cephadm disks, which is making it impossible to access our HPC Cluster: # ceph health detail HEALTH_WARN 3 failed cephadm daemon(s); insufficient standby MDS daemons available [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) daemon mds.cephfs.s1.nvopyf on s1.ceph.infra.ufscar.br is in error state daemon mds.cephfs.s2.qikxmw on s2.ceph.infra.ufscar.br is in error state daemon mds.cftv.s2.anybzk on s2.ceph.infra.ufscar.br is in error state [WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available have 0; want 1 more Googling we found out that we should remove the failed MDS, but the data in these disks is relatively important. We would like to know if we need to remove it or if it can be fixed, and if we have to remove it if the data will be lost. Please tell me if you need more information. Thanks in advance, André de Freitas Smaira Federal University of São Carlos - UFSCar

3 3

Re: Telemetry service is temporarily down
by Yaarit Hatuka 14 Jan '23

14 Jan '23

Hi everyone, Our telemetry service is up and running again. Thanks Adam Kraitman and Dan Mick for restoring the service. We thank you for your patience and appreciate your contribution to the project! Thanks, Yaarit On Tue, Jan 3, 2023 at 3:14 PM Yaarit Hatuka <yhatuka(a)redhat.com> wrote: > Hi everyone, > > We are having some infrastructure issues with our telemetry backend, and > we are working on fixing it. > Thanks Jan Horacek for opening this issue > <https://tracker.ceph.com/issues/58371> [1]. We will update once the > service is back up. > We are sorry for any inconvenience you may be experiencing, and appreciate > your patience. > > Thanks, > Yaarit > > [1] https://tracker.ceph.com/issues/58371 >

1 0

Current min_alloc_size of OSD?
by Robert Sander 13 Jan '23

13 Jan '23

Hi, Ceph 16 Pacific introduced a new smaller default min_alloc_size of 4096 bytes for HDD and SSD OSDs. How can I get the current min_allloc_size of OSDs that were created with older Ceph versions? Is there a command that shows this info from the on disk format of a bluestore OSD? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

5 7

Problem with IO after renaming File System .data pool
by Murilo Morais 13 Jan '23

13 Jan '23

Good morning everyone. That night we went through an accident, where they accidentally renamed the .data pool of a File System making it instantly inaccessible, when renaming it again to the correct name it was possible to mount and list the files, but could not read or write. When trying to write, the FS returned as Read Only, when trying to read it returned Operation not allowed. After a period of breaking my head I tried to mount with the ADMIN user and everything worked correctly. I tried to remove the authentication of the current user through `ceph auth rm`, I created a new user through `ceph fs authorize <fs_name> client.<user> / rw` and it continued the same way, I also tried to recreate it through `ceph auth get-or-create` and nothing different happened, it stayed exactly the same. After setting `allow *` in mon, mds and osd I was able to mount, read and write again with the new user. I can understand why the File System stopped after renaming the pool, what I don't understand is why users are unable to perform operations on FS even with RW or any other user created. What could have happened behind the scenes to not be able to perform IO even with the correct permissions? Or did I apply incorrect permissions that caused this problem? Right now everything is working, I would really like to understand what happened, because I didn't find anything documented about this type of incident.

1 0

radosgw ceph.conf question
by Boris Behrens 13 Jan '23

13 Jan '23

Hi, I am just reading through this document ( https://docs.ceph.com/en/octopus/radosgw/config-ref/) and on the top is states: The following settings may added to the Ceph configuration file (i.e., > usually ceph.conf) under the [client.radosgw.{instance-name}] section. > And my ceph.conf looks like this: [client.eu-central-1-s3db3] > rgw_frontends = beast endpoint=[::]:7482 > rgw_region = eu > rgw_zone = eu-central-1 > > [client.eu-central-1-s3db3-old] > rgw_frontends = beast endpoint=[::]:7480 > rgw_region = eu > rgw_zone = eu-central-1 > > [client.eu-customer-1-s3db3] > rgw_frontends = beast endpoint=[::]:7481 > rgw_region = eu-someother > rgw_zone = eu-someother-1 > Do I need to change the section names? It also seems that rgw_region is a non-existing config value (this might have come from very old RHCS documentation) Would be very nice if someone could help me clarify this. Cheers and happy weekend Boris

1 0