April 2023 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

Re: Ceph Mgr/Dashboard Python depedencies: a new approach
by Casey Bodley 14 Jul '23

14 Jul '23

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

4 10

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 04 Jul '23

04 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

RadosGW S3 API Multi-Tenancy
by Brad House 08 Jun '23

08 Jun '23

I'm in the process of exploring if it is worthwhile to add RadosGW to our existing ceph cluster. We've had a few internal requests for exposing the S3 API for some of our business units, right now we just use the ceph cluster for VM disk image storage via RBD. Everything looks pretty straight forward until we hit multitenancy. The page on multi-tenancy doesn't dive into permission delegation: https://docs.ceph.com/en/quincy/radosgw/multitenancy/ The end goal I want is to be able to create a single user per tenant (Business Unit) which will act as their 'administrator', where they can then do basically whatever they want under their tenant sandbox (though I don't think we need more advanced cases like creations of roles or policies, just create/delete their own users, buckets, objects). I was hopeful this would just work, and I asked on the ceph IRC channel on OFTC and was told once I grant a user caps="users=*", they would then be allowed to create users *outside* of their own tenant using the Rados Admin API and that I should explore IAM roles. I think it would make sense to add a feature, such as a flag that can be set on a user, to ensure they stay in their "sandbox". I'd assume this is probably a common use-case. Anyhow, if its possible to do today using iam roles/policies, then great, unfortunately this is my first time looking at this stuff and there are some things not immediately obvious. I saw this online about AWS itself and creating a permissions boundary, but that's for allowing creation of roles within a boundary: https://www.qloudx.com/delegate-aws-iam-user-and-role-creation-without-givi… I'm not sure what "Action" is associated with the Rados Admin API create user for applying a boundary that the user can only create users with the same tenant name. https://docs.ceph.com/en/quincy/radosgw/adminops/#create-user Any guidance on this would be extremely helpful. Thanks! -Brad

1 1

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

ln: failed to create hard link 'file name': Read-only file system
by Frank Schilder 26 May '23

26 May '23

Hi all, on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm un-taring a larger package (1.2G) and after some time I get these errors: ln: failed to create hard link 'file name': Read-only file system The strange thing is that this seems only temporary. When I used "ln src dst" for manual testing, the command failed as above. However, after that I tried "ln -v src dst" and this command created the hard link with exactly the same path arguments. During the period when the error occurs, I can't see any FS in read-only mode, neither on the NFS client nor the NFS server. Funny thing is that file creation and write still works, its only the hard-link creation that fails. For details, the set-up is: file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server other server: mount /shares/path as NFS More precisely, on the file-server: fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0 exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP On the host at DEST-IP: fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0 Both, the file server and the client server are virtual machines. The file server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64). When I change the NFS export from "async" to "sync" everything works. However, that's a rather bad workaround and not a solution. Although this looks like an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a race with scheduling and executing operations on the ceph-fs kernel mount. Has anyone seen something like that? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 13

Radosgw multisite replication issues
by Tarrago, Eli (RIS-BCT) 12 May '23

12 May '23

Good Afternoon, I am experiencing an issue where east-1 is no longer able to replicate from west-1, however, after a realm pull, west-1 is now able to replicate from east-1. In other words: West <- Can Replicate <- East West -> Cannot Replicate -> East After confirming the access and secret keys are identical on both sides, I restarted all radosgw services. Here is the current status of the cluster below. Thank you for your help, Eli Tarrago root@east01:~# radosgw-admin zone get { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "domain_root": "rgw-east.rgw.meta:root", "control_pool": "rgw-east.rgw.control", "gc_pool": "rgw-east.rgw.log:gc", "lc_pool": "rgw-east.rgw.log:lc", "log_pool": "rgw-east.rgw.log", "intent_log_pool": "rgw-east.rgw.log:intent", "usage_log_pool": "rgw-east.rgw.log:usage", "roles_pool": "rgw-east.rgw.meta:roles", "reshard_pool": "rgw-east.rgw.log:reshard", "user_keys_pool": "rgw-east.rgw.meta:users.keys", "user_email_pool": "rgw-east.rgw.meta:users.email", "user_swift_pool": "rgw-east.rgw.meta:users.swift", "user_uid_pool": "rgw-east.rgw.meta:users.uid", "otp_pool": "rgw-east.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-east.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-east.rgw.buckets.data" } }, "data_extra_pool": "rgw-east.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-east.rgw.log:notif" } root@west01:~# radosgw-admin zone get { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "domain_root": "rgw-west.rgw.meta:root", "control_pool": "rgw-west.rgw.control", "gc_pool": "rgw-west.rgw.log:gc", "lc_pool": "rgw-west.rgw.log:lc", "log_pool": "rgw-west.rgw.log", "intent_log_pool": "rgw-west.rgw.log:intent", "usage_log_pool": "rgw-west.rgw.log:usage", "roles_pool": "rgw-west.rgw.meta:roles", "reshard_pool": "rgw-west.rgw.log:reshard", "user_keys_pool": "rgw-west.rgw.meta:users.keys", "user_email_pool": "rgw-west.rgw.meta:users.email", "user_swift_pool": "rgw-west.rgw.meta:users.swift", "user_uid_pool": "rgw-west.rgw.meta:users.uid", "otp_pool": "rgw-west.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-west.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-west.rgw.buckets.data" } }, "data_extra_pool": "rgw-west.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-west.rgw.log:notif" east01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "init", "num_shards": 0, "period": "", "realm_epoch": 0 }, "markers": [] }, "full_sync": { "total": 0, "complete": 0 } } west01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "sync", "num_shards": 64, "period": "44b6b308-e2d8-4835-8518-c90447e7b55c", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 46, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, #### goes on for a long time… { "key": 63, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } } ] }, "full_sync": { "total": 46, "complete": 46 } } east01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) metadata sync no sync (zone is master) 2023-04-20T19:03:13.388+0000 7f25fa036c80 0 ERROR: failed to fetch datalog info data sync source: b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) failed to retrieve sync info: (13) Permission denied west01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 16 shards behind shards: [5,56,62,65,66,70,76,86,87,94,104,107,111,113,120,126] oldest incremental change not applied: 2023-04-20T19:02:48.783283+0000 [5] east01:~# radosgw-admin zonegroup get { "id": "0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd", "name": "EastWestceph", "api_name": "EastWestceph", "is_master": "true", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080, http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "hostnames": [ "eastvip.example.net", "westvip.example.net" ], "hostnames_s3website": [], "master_zone": "ddd66ab8-0417-46ee-a53b-043352a63f93", "zones": [ { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "endpoints": [ http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "sync_policy": { "groups": [] } } ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

4 6

Lua scripting in the rados gateway
by Thomas Bennett 11 May '23

11 May '23

Hi ceph users, I've been trying out the lua scripting for the rados gateway (thanks Yuval). As in my previous email I mentioned that there is an error when trying to load the luasocket module. However, I thought it was a good time to report on my progress. My 'hello world' example below is called *test.lua* below includes the following checks: 1. Can I write to the debug log? 2. Can I use the lua socket package to do something stupid but intersting, like connect to a webservice? Before you continue reading this, you might need to know that I run all ceph processes in a *CentOS Stream release 8 *container deployed using ceph orchestrator running *Ceph v17.2.5*, so please view the information below in that context. For anyone looking for a reference, I suggest going to the ceph lua rados gateway documentation at radosgw/lua-scripting <https://docs.ceph.com/en/quincy/radosgw/lua-scripting/>. There are two new switches you need to know about in the radosgw-admin: - *script* -> loading your lua script - *script-package* -> loading supporting packages for your script - e.i. luasocket in this case. For a basic setup, you'll need to have a few dependencies in your containers: - cephadm container: requires luarocks (I've checked the code - it runs a luarocks search command) - radosgw container: requires luarocks, gcc, make, m4, wget (wget just in case). To achieve the above, I updated the container image for our running system. I needed to do this because I needed to redeploy the rados gateway container to inject the lua script packages into the radosgw runtime process. This will start with a fresh container based on the global config *container_image* setting on your running system. For us this is currently captured in *quay.io/tsolo/ceph:v17.2.5-3 <http://quay.io/tsolo/ceph:v17.2.5-3>* and included the following exta steps (including installing the lua dev from an rpm because there is no centos package in yum): yum install luarocks gcc make wget m4 rpm -i https://rpmfind.net/linux/centos/8-stream/PowerTools/x86_64/os/Packages/lua… You will notice that I've included a compiler and compiler support into the image. This is because luarocks on the radosgw to compile luasocket (the package I want to install). This will happen at start time when the radosgw is restarted from ceph orch. In the cephadm container I still need to update our cephadm shell so I need to install luarocks by hand: yum install luarocks Then set thew updated image to use: ceph config set global container_image quay.io/tsolo/ceph:v17.2.5-3 I now create a file called: *test.lua* in the cephadm container. This contains the following lines to write to the log and then do a get request to google. This is not practical in production, but it serves the purpose of testing the infrastructure: RGWDebugLog("Tsolo start lua script") local LuaSocket = require("socket") client = LuaSocket.connect("google.com", 80) client:send("GET / HTTP/1.0\r\nHost: google.com\r\n\r\n") while true do s, status, partial = client:receive('*a') RGWDebugLog(s or partial) if status == "closed" then break end end client:close() RGWDebugLog("Tsolo stop lua") Next I run: radosgw-admin script-package add --package=luasocket --allow-compilation And then list the added package to make sure it is there: radosgw-admin script-package list Note - at this point the radosgw has not been modified, it must first be restarted. Then I put the *test.lua *script into the pre request context: radosgw-admin script put --infile=test.lua --context=preRequest You also need to raise the debug log level on the running rados gateway: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.asok config set debug_rgw 20 Inside the radosgw container I apply my fix (as per previous email): cp -ru /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib64/* /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib/ Outside on the host running the radosgw-admin container I follow the journalctl for the radosgw container (to get the logs): journalctl -fu ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@rgw. xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.service Then I run an s3cmd to put data in via the rados gateway and check the journalctl logs and see: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua So the script worked :) If you want to see where the luarocks libraries have been installed, look in the /tmp/ directory of the radosgw container after you redeploy it and you will find the content in /tmp/luarocks. Conclusions: There was a bit to figure out to get this working, but now that I've got this simple test working I think there is a lot more to look into and discover and use w.r.t. this powerful tool. Cheers, Tom

2 5

How can I use not-replicated pool (replication 1 or raid-0)
by mhnx 10 May '23

10 May '23

Hello. I have a 10 node cluster. I want to create a non-replicated pool (replication 1) and I want to ask some questions about it: Let me tell you my use case: - I don't care about losing data, - All of my data is JUNK and these junk files are usually between 1KB to 32MB. - These files will be deleted in 5 days. - Writable space and I/O speed is more important. - I have high Write/Read/Delete operations, minimum 200GB a day. I'm afraid that, in any failure, I won't be able to access the whole cluster. Losing data is okay but I have to ignore missing files, remove the data from the cluster and continue with existing data and while doing this, I want to be able to write new data to the cluster. My questions are: 1- To reach this goal do you have any recommendations? 2- With this setup, what potential problems do you have in mind? 3- I think Erasure Coding is not a choice because of the performance problems and slow file deletion. With this I/O need EC will miss files and leaks may happen (I've seen before on Nautilus). 4- You read my needs, is there a better way to do this? Thank you for the answers. Best regards.

3 8

ERROR: Distro uos version 20 not supported
by Ben 09 May '23

09 May '23

Hi, This seems not very relevant since all ceph components are running in containers. Any ideas to get over this issue? Any other ideas or suggestions on this kind of deployment? sudo ./cephadm --image 10.21.22.1:5000/ceph:v17.2.5-20230316 --docker bootstrap --mon-ip 10.21.22.1 --skip-monitoring-stack Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... No time sync service is running; checked for ['chrony.service', 'chronyd.service', 'systemd-timesyncd.service', 'ntpd.service', 'ntp.service', 'ntpsec.service', 'openntpd.service'] ERROR: Distro uos version 20 not supported uname -a Linux aaaa 4.19.0-91.82.42.uelc20.x86_64 #1 SMP Sat May 15 13:50:04 CST 2021 x86_64 x86_64 x86_64 GNU/Linux Thank you in advance Ben

1 1

16.2.13 pacific QE validation status
by Yuri Weinstein 08 May '23

08 May '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59542#note-1 Release Notes - TBD Seeking approvals for: smoke - Radek, Laura rados - Radek, Laura rook - Sébastien Han cephadm - Adam K dashboard - Ernesto rgw - Casey rbd - Ilya krbd - Ilya fs - Venky, Patrick upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8) upgrade/pacific-p2p - Laura powercycle - Brad (SELinux denials) ceph-volume - Guillaume, Adam K Thx YuriW

10 26

Frequent calling monitor election
by Frank Schilder 04 May '23

04 May '23

Hi all, our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting: 2/9/23 4:42:30 PM[INF]overall HEALTH_OK 2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:40:00 PM[INF]overall HEALTH_OK 2/9/23 4:30:00 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:04 PM[INF]overall HEALTH_OK 2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:20:00 PM[INF]overall HEALTH_OK 2/9/23 4:10:00 PM[INF]overall HEALTH_OK 2/9/23 4:00:00 PM[INF]overall HEALTH_OK 2/9/23 3:50:00 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again? Thanks for any help! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 9

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

RBD mirroring, asking for clarification
by wodel youchi 03 May '23

03 May '23

Hi, When using rbd mirroring, the mirroring concerns the images only, not the whole pool? So, we don't need to have a dedicated pool in the destination site to be mirrored, the only obligation is that the mirrored pools must have the same name. In other words, We create two pools with the same name, one on the source site the other on the destination site, we create the mirror link (one way or two ways replication), then we choose what images to sync. Both pools can be used simultaneously on both sites, it's the mirrored images that cannot be used simultaneously, only promoted ones. Is this correct? Regards.

2 5

architecture help (iscsi, rbd, backups?)
by Angelo Höngens 02 May '23

02 May '23

Hey guys and girls, I'm working on a project to build storage for one of our departments, and I want to ask you guys and girls for input on the high-level overview part. It's a long one, I hope you read along and comment. SUMMARY I made a plan last year to build a 'storage solution' including ceph and some windows VM's to expose the data over SMB to clients. A year later I finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph itself runs great, but when I wanted to start exposing the data using iscsi to our VMware farm, I ran into some issues. I know the iscsi gateways will introduce some new performance bottlenecks, but I'm seeing really slow performance, still working on that. But then I ran into the warning on the iscsi gateway page: "The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features.". Wait, what? Why!? What does this mean? Does this mean that iSCSI is now 'feature complete' and will still be supported the next 5 years, or will it be deprecated in the future? I tried searching, but couldn't find any info on the decision and the roadmap. My goal is to build a future-proof setup, and using deprecated components should not be part of that of course. If the iscsi gateway will still be supported the next few years and I can iron out the performance issues, I can still go on with my original plan. If not, I have to go back to the drawing board. And maybe you guys would advise me to take another route anyway. GOALS My goals/considerations are: - we want >1PB of storage capacity for cheap (on a tight budget) for research data. Most of it is 'store once, read sometimes'. <1% of the data is 'hot'. - focus is on capacity, but it would be nice to have > 200MB/s of sequential write/read performance and not 'totally suck' on random i/o. Yes, not very well quantified, but ah. Sequential writes are most important. - end users all run Windows computers (mostly VDI's) and a lot of applications require SMB shares. - security is a big thing, we want really tight ACL's, specific monitoring agents, etc. - our data is incredibly important to us, we still want the 3-2-1 backup rule. Primary storage solution, a second storage solution in a different place, and some of the data that is not reproducible is also written to tape. We also want to be protected from ransomware or user errors (so no direct replication to the second storage). - I like open source, reliability, no fork-lift upgrades, no vendor lock-in, blah, well, I'm on the ceph list here, no need to convince you guys ;) - We're hiring a commercial company to do ceph maintenance and support for when I'm on leave or leaving the company, but they won't support clients, backup software, etc, so I want something as simple as possible. We do have multiple Windows/VMware admins, but no other real linux guru's. THE INITIAL PLAN Given these considerations, I ordered two identical clusters, each consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and 10 capacity disks (EC 4:2 for the data), and each node is connected using a 2x25Gbps bond. Ceph is running like a charm. Now I just have to think about exposing the data to end users, and I've been testing different setups. My original plan was to expose for example 10x100TB rbd images using iSCSI to our VMware farm, formatting the luns with VMFS6, and run for example 2 Windows file servers per datastore on that with a single DFS namespace to end users. Then backup the file servers using our existing Veeam infrastructure to RGW running on the second cluster with an immutable bucket. This way we would have easily defined security boundaries: the clients can only reach the file servers, the file servers only see their local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file server would be compromised, it would have no access to ceph. We have easy incremental backups, immutability for ransomware protection, etc. And the best part is that the ceph admin can worry about ceph, the vmware admin can focus on ESX, VMFS and all the vmware stuff, and the Windows admins can focus on the Windows boxes, Windows-specific ACLS and tools and Veeam backups and stuff. CURRENT SITUATION I'm building out this plan now, but I'm running into issues with iSCSI. Are any of you doing something similar? What is your iscsi performance compared to direct rbd? In regard to performance: If I take 2 test windows VM's, I put one on an iSCSI datastore and another with direct rbd access using the windows rbd driver, I create a share on those boxes and push data to it, I see different results (of course). Copying some iso images over SMB to the 'windows vm running direct rbd' I see around 800MB/s write, and 200MB/s read, which is pretty okay. When I send data to the 'windows vm running on top of iscsi' it starts writing at around 350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really acceptable. Another really viable and performant scenario would be to have the Windows file servers connect to rbd directly with the windows rbd driver. It seems to work well, it's fast, and you don't have the bottleneck that the iscsi gateway creates. But I see this driver is still in beta. Is anyone using this in production? What are your experiences? We would miss out on the separation of layers and thus have less security, but at the same time, it really increases efficiency and performance. And if I use rbd, then vmware won't see the storage, and I cannot do an image backup using veeam. I could of course do backups of the rbd images, using tools like restic or backy to rgw running on the second cluster with immutable buckets. What are your experiences? Is it easy to do differential backups of lots of 50TB rbd images? Change rate is usually like 0.005% per day or something ;) By the way, we also thought about CephFS, but we have some complex stuff going on with extended ACL's that I don't think will play nice with CephFS, and I think it's a lot more complex to backup CephFS than block images. If you made it here, thank you for your time! I hope you can share thoughts on my questions! Angelo.

6 8

Memory leak in MGR after upgrading to pacific.
by Gary Molenkamp 02 May '23

02 May '23

Good morning, After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days ago, I'm noticing that the MGR daemons keep failing over to standby and then back every 24hrs. Watching the output of 'ceph orch ps' I can see that the memory consumption of the mgr is steadily growing until it becomes unresponsive. When the mgr becomes unresponsive, tasks such as RESTful calls start to fail, and the standby eventually takes over after ~20 minutes. I've included a log of memory consumption (in 10 minute intervals) at the end of this message. While the cluster recovers during this issue, the loss of usage data during the outage, and the fact its occurring is problematic. Any assistance would be appreciated. Note, this is a cluster that has been upgraded from an original jewel based ceph using filestore, through bluestore conversion, container conversion, and now to Pacific. The data below shows memory use with three mgr modules enabled: cephadm, restful, iostat. By disabling iostat, I can reduce the rate of memory consumption increasing to about 200MB/hr. Thanks Gary. Wed Apr 26 12:50:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 7m ago 3h 1099M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 7m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:00:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1161M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:10:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1230M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:20:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 6m ago 4h 1250M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:30:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1318M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 5m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:40:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:50:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1442M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:00:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1498M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:10:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1554M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:20:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 4m ago 5h 1617M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:30:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1677M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:40:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1735M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 3m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:50:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1806M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:00:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:10:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1920M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:20:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 114s ago 6h 1981M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 113s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:30:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 95s ago 6h 2048M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 95s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:40:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 80s ago 6h 2121M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 80s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:50:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 53s ago 6h 2170M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 53s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:00:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 35s ago 6h 2238M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 34s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:10:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 19s ago 6h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 19s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:20:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 10m ago 7h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 10m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:30:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2321M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 9m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:40:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2382M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:50:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2449M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:00:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2505M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:10:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2562M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:20:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 8m ago 8h 2628M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:30:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2692M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 7m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:40:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2761M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:50:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2811M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:00:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:10:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2923M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:20:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 6m ago 9h 2995M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:30:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3055M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 5m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:40:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3110M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:50:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3166M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:00:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3244M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:10:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3289M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:20:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 4m ago 10h 3368M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:30:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3437M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 3m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:40:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3496M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:50:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3546M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:00:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3627M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:10:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3637M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:20:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 92s ago 11h 492M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 91s ago 23h 418M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:31:55 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 51s ago 11h 373M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 51s ago 23h 520M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:41:56 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 32s ago 11h 374M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 32s ago 0h 585M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:51:57 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 17s ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 17s ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:01:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 10m ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 10m ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:11:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 9m ago 11h 378M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 698M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:21:59 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 758M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:32:00 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 825M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:42:01 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 895M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:52:02 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 948M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:02:03 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 1010M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:12:04 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 7m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1071M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:22:05 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1132M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:32:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1183M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:42:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1244M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:52:07 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1311M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:02:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1380M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:12:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 5m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1441M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:22:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1508M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:32:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1541M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:42:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1610M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:52:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1674M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:02:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1725M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:12:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 3m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1794M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:22:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1844M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:32:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1917M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:42:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2002M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:52:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2056M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:02:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2113M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:12:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 92s ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 91s ago 4h 2173M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:22:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 73s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 72s ago 4h 2201M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:32:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 57s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 57s ago 4h 2264M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:42:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 29s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 29s ago 5h 2324M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:52:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10s ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:02:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10m ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10m ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:12:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 9m ago 16h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2442M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:22:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2502M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:32:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2552M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:42:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2614M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:52:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2681M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:02:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2748M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:12:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 7m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2810M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:22:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2867M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:32:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2916M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:42:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3024M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:52:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3070M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:02:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3127M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:12:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 5m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3183M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:22:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 5m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3215M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:32:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 4m ago 7h 3290M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:42:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3342M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:52:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3392M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:02:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3451M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:12:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3516M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:22:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 3m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3566M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:32:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 2m ago 8h 3632M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:42:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3696M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:52:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3760M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:02:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 115s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 114s ago 9h 3813M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:12:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 99s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 99s ago 9h 3869M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:22:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 72s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 72s ago 9h 3923M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:32:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 54s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 53s ago 9h 3990M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:42:49 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 38s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 38s ago 10h 4040M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:52:50 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 11s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10s ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:02:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 10m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10m ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:12:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 9m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4157M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:22:52 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (22h) 9m ago 22h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4213M - 16.2.12 921c4e969fff 6fed006895e8 -- Gary Molenkamp Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

1 1

PVE CEPH OSD heartbeat show
by Peter 02 May '23

02 May '23

Dear all, We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues. Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU. When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes. We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline. We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,

3 4

Re: cephfs - max snapshot limit?
by MARTEL Arnaud 02 May '23

02 May '23

Hi Venky, > Also, at one point the kclient wasn't able to handle more than 400 snapshots (per file system), but we have come a long way from that and that is not a constraint right now. Does it mean that there is no more limit to the number of snapshots per filesystem? And, if not, do you know what is the max number of snapshots now (per filesystem) ?? Cheers, Arnaud

2 1

Ceph recovery
by wodel youchi 01 May '23

01 May '23

Hi, When creating a ceph cluster, a failover domain is created, and by default it uses host as a minimal domain, that domain can be modified to chassis, or rack, ...etc. My question is : Suppose I have three osd nodes, my replication is 3 and my failover domain is host, which means that each copy of data is stored on a different node. What happens when one node crashes, does Ceph use the remaining free space on the other two to create the third copy, or the ceph cluster will run in degraded mode, like a RAID5 which lost a disk. Regards.

2 2

Nearly 1 exabyte of Ceph storage
by Yaarit Hatuka 01 May '23

01 May '23

Hi everyone, We are excited to share with you the latest statistics from our Ceph public telemetry dashboards <https://telemetry-public.ceph.com/>. With almost 2,500 clusters reporting nearly 1 exabyte of total storage capacity, we can see a steady growth rate <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=19&orgId=…> since the telemetry service was launched. One of the things telemetry helps us to understand is version adoption rate. See, for example, the trend of Quincy <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=28&orgId=…> deployments in the community. We can also learn that the average cluster size grows over time <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=33&orgId=1>. Ceph telemetry is on an opt-in basis. You can opt-in with: `ceph telemetry on` Learn more here <https://docs.ceph.com/en/latest/mgr/telemetry/#enabling-telemetry>. Help us cross the exabyte mark by opting-in today! Learn more about the latest developments around Telemetry <https://sched.co/1JKZ2> at the upcoming Cephalocon. Thank you for your contribution, Yaarit [image: ceph_telemetry_dashboards.jpg]

2 3

RGW Lua - cancel request
by Ondřej Kukla 01 May '23

01 May '23

Hello, Lately I’ve been playing with Lua scripting on top of RGW. I would like to implement a request blocking based on bucket name -> when there is a dot in a bucket name return error code and a message that this name is invalid. Here is the code I was able to came up with. if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = “InvalidBucketName" Request.Response.Message = “Dots in bucket name are not allowed." end This works fine, but the request for creating a bucket would be processed and the bucket will be created. I thought about a dirty workaround with setting the Request.Bucket.Name to a bucket that already exists but it seems that this field is not writable in Quincy. Is there a way to block the request from processing? Any help is much appreciated. Kind regards, Ondrej

3 3

backing up CephFS
by Angelo Höngens 30 Apr '23

30 Apr '23

How do you guys backup CephFS? (if at all?) I'm building 2 ceph clusters, a primary one and a backup one, and I'm looking into CephFS as the primary store for research files. CephFS mirroring seems a very fast and efficient way to copy data to the backup location, and it has the benefit of the files on the backup location being fully in a ready-to-use state instead of some binary proprietary archive. But I am wondering how to do 'ransomware protection' in this setup. I can't believe I'm the only one that wants to secure my data ;) I'm reading up on snapshots and mirroring, and that's great to protect from user error. I could schedule snapshots on the primary cluster, and they would automatically get synced to the backup cluster. But a user can still delete all snapshots on the source side, right? And you need to create a ceph user on the backup cluster, and import that on the primary cluster. That means that if a hacker has those credentials, he could also delete the data on the backup cluster? Or is there some 'append-only' mode for immutability? Another option I'm looking into is restic. Restic looks like a cool tool, but it does not support s3 object locks yet. See the discussion here [1]. I should be able to get immutability working with the restic-rest backend according to the developer. But I have my worries that running restic to sync up an 800TB filesystem with millions of files will be.. worrysome ;) Anyone using restic in production? Thanks again for your input! Angelo. [1] https://github.com/restic/restic/issues/3195

4 3

Deep-scrub much slower than HDD speed
by Niklas Hambüchen 30 Apr '23

30 Apr '23

I observed that on an otherwise idle cluster, scrubbing cannot fully utilise the speed of my HDDs. `iostat` shows only 8-10 MB/s per disk, instead of the ~100 MB/s most HDDs can easily deliver. Changing scrubbing settings does not help (see below). Environment: * 6 active+clean+scrubbing+deep * Ceph version 16.2.7. * BlueStore * My cluster has many objects small objects ("402.32M objects, 38 TiB" from "ceph status") due to small files (4 - 32 KiB) on CephFS. `iostat -x 5` with default settings: Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 198.60 6878.40 0.00 0.00 12.78 34.63 51.80 2612.80 0.00 0.00 14.82 50.44 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.30 91.24 dm-1 0.80 3.20 0.00 0.00 11.50 4.00 52.60 2582.40 0.00 0.00 13.69 49.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.73 3.78 dm-10 11.20 71.20 0.00 0.00 0.09 6.36 145.80 583.20 0.00 0.00 0.14 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 2.62 dm-11 192.60 6737.60 0.00 0.00 10.74 34.98 34.80 1684.80 0.00 0.00 11.47 48.41 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.47 91.40 dm-12 245.40 10194.40 0.00 0.00 9.43 41.54 21.20 575.20 0.00 0.00 3.94 27.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.40 87.92 dm-13 30.80 1772.80 0.00 0.00 11.61 57.56 78.80 4507.20 0.00 0.00 19.78 57.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.92 9.54 dm-14 3.20 24.80 0.00 0.00 0.12 7.75 125.20 500.80 0.00 0.00 0.12 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.18 dm-15 2.80 19.20 0.00 0.00 0.14 6.86 105.40 421.60 0.00 0.00 0.05 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.76 dm-16 0.80 6.40 0.00 0.00 0.00 8.00 111.00 444.00 0.00 0.00 0.10 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.82 dm-17 10.80 76.80 0.00 0.00 0.09 7.11 151.40 605.60 0.00 0.00 0.08 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.92 dm-18 10.20 67.20 0.00 0.00 0.08 6.59 115.60 462.40 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.16 dm-19 10.20 56.80 0.00 0.00 0.10 5.57 109.00 436.00 0.00 0.00 0.07 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.34 dm-2 4.80 435.20 0.00 0.00 0.12 90.67 751.80 6292.80 0.00 0.00 0.07 8.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 16.14 dm-20 0.40 2.40 0.00 0.00 0.00 6.00 265.00 2459.20 0.00 0.00 0.10 9.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 5.36 dm-21 191.00 6105.60 0.00 0.00 6.34 31.97 67.80 3748.00 0.00 0.00 19.56 55.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.54 42.22 dm-3 1.00 8.80 0.00 0.00 0.00 8.80 91.00 364.00 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 dm-4 167.60 4973.60 0.00 0.00 10.15 29.68 49.20 2511.20 0.00 0.00 11.39 51.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.26 89.18 dm-5 11.20 73.60 0.00 0.00 0.12 6.57 124.40 497.60 0.00 0.00 0.07 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.16 dm-6 27.20 1644.80 0.00 0.00 12.22 60.47 57.20 3316.80 0.00 0.00 15.93 57.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.24 6.78 dm-7 217.40 8032.80 0.00 0.00 12.04 36.95 64.40 3654.40 0.00 0.00 23.69 56.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.14 97.28 dm-8 10.80 70.40 0.00 0.00 0.15 6.52 111.80 447.20 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.32 dm-9 1.60 8.00 0.00 0.00 13.25 5.00 46.60 2563.20 0.00 0.00 9.08 55.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.44 3.70 md127 1.60 107.20 0.00 0.00 0.00 67.00 142.00 1856.00 0.00 0.00 0.05 13.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.80 nvme0n1 42.00 772.00 0.00 0.00 0.10 18.38 1001.00 10503.50 451.20 31.07 0.03 10.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 25.20 nvme1n1 37.20 248.00 0.00 0.00 0.09 6.67 609.80 6723.50 369.00 37.70 0.04 11.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 15.80 sda 0.80 3.20 0.00 0.00 11.50 4.00 33.60 2582.40 19.00 36.12 8.43 76.86 0.00 0.00 0.00 0.00 0.00 0.00 3.20 0.25 0.29 3.78 sdb 117.80 10195.20 127.80 52.04 11.53 86.55 18.80 575.20 2.40 11.32 3.93 30.60 0.00 0.00 0.00 0.00 0.00 0.00 2.80 3.93 1.44 87.90 sdc 4.60 1644.80 22.60 83.09 11.74 357.57 23.60 3316.80 33.60 58.74 8.53 140.54 0.00 0.00 0.00 0.00 0.00 0.00 1.80 4.56 0.26 6.74 sdd 109.40 4975.20 58.40 34.80 12.19 45.48 22.00 2511.20 27.20 55.28 5.75 114.15 0.00 0.00 0.00 0.00 0.00 0.00 2.80 6.07 1.48 89.16 sde 115.40 6563.20 77.00 40.02 12.21 56.87 26.20 1684.80 8.60 24.71 7.82 64.31 0.00 0.00 0.00 0.00 0.00 0.00 1.80 6.78 1.63 91.36 sdf 6.40 1772.80 24.40 79.22 14.84 277.00 39.20 4507.20 39.60 50.25 13.93 114.98 0.00 0.00 0.00 0.00 0.00 0.00 3.20 2.25 0.65 9.54 sdg 121.60 8033.60 95.80 44.07 12.39 66.07 30.60 3654.40 33.80 52.48 12.47 119.42 0.00 0.00 0.00 0.00 0.00 0.00 2.60 8.77 1.91 97.24 sdh 122.00 6105.60 69.00 36.13 5.47 50.05 33.00 3748.00 34.80 51.33 16.94 113.58 0.00 0.00 0.00 0.00 0.00 0.00 2.20 5.00 1.24 42.20 sdi 117.00 6856.80 81.60 41.09 12.25 58.61 32.00 2612.80 19.80 38.22 10.80 81.65 0.00 0.00 0.00 0.00 0.00 0.00 2.60 8.46 1.80 91.18 sdj 1.60 8.00 0.00 0.00 13.12 5.00 31.20 2563.20 15.40 33.05 5.06 82.15 0.00 0.00 0.00 0.00 0.00 0.00 2.00 1.70 0.18 3.70 With settings osd_deep_scrub_stride = 4194304 osd_scrub_load_threshold = 20 osd_scrub_chunk_min = 15 osd_scrub_chunk_max = 75 osd_max_scrubs = 3 I get a slight improvement only: Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 400.60 14686.40 0.00 0.00 19.93 36.66 25.20 1197.60 0.00 0.00 27.44 47.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.67 91.32 dm-1 362.60 10583.20 0.00 0.00 18.58 29.19 30.40 1742.40 0.00 0.00 23.32 57.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.45 98.14 dm-10 6.40 64.00 0.00 0.00 0.03 10.00 37.60 150.40 0.00 0.00 0.06 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.02 dm-11 76.60 1939.20 0.00 0.00 10.61 25.32 35.20 1885.60 0.00 0.00 22.32 53.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.60 29.06 dm-12 93.00 4178.40 0.00 0.00 16.04 44.93 4.40 96.00 0.00 0.00 0.68 21.82 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.50 19.04 dm-13 376.80 12716.00 0.00 0.00 15.90 33.75 27.20 1444.80 0.00 0.00 23.76 53.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.64 97.76 dm-14 11.80 312.80 0.00 0.00 0.12 26.51 33.40 133.60 0.00 0.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 dm-15 7.00 430.40 0.00 0.00 0.17 61.49 46.20 184.80 0.00 0.00 0.06 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.94 dm-16 11.80 229.60 0.00 0.00 0.10 19.46 49.20 196.80 0.00 0.00 0.09 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.20 dm-17 4.00 39.20 0.00 0.00 0.10 9.80 29.60 118.40 0.00 0.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.72 dm-18 9.00 94.40 0.00 0.00 0.07 10.49 37.20 148.80 0.00 0.00 0.30 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.92 dm-19 7.40 70.40 0.00 0.00 0.11 9.51 56.80 227.20 0.00 0.00 0.02 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.04 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 62.00 482.40 0.00 0.00 0.07 7.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.16 dm-20 12.40 2375.20 0.00 0.00 0.23 191.55 314.20 1524.80 0.00 0.00 0.03 4.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 7.90 dm-21 321.00 9468.00 0.00 0.00 7.86 29.50 31.20 1630.40 0.00 0.00 11.69 52.26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.89 42.84 dm-3 12.40 125.60 0.00 0.00 0.08 10.13 37.00 148.00 0.00 0.00 0.02 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.84 dm-4 374.00 13150.40 0.00 0.00 18.31 35.16 23.60 1100.80 0.00 0.00 12.42 46.64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.14 96.70 dm-5 8.00 84.00 0.00 0.00 0.10 10.50 38.00 152.00 0.00 0.00 0.03 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 dm-6 201.60 6619.20 0.00 0.00 11.56 32.83 3.40 108.80 0.00 0.00 2.29 32.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.34 58.24 dm-7 414.40 14476.00 0.00 0.00 21.99 34.93 10.80 235.20 0.00 0.00 5.19 21.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.17 98.00 dm-8 0.60 14.40 0.00 0.00 0.00 24.00 40.80 163.20 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.70 dm-9 478.00 17300.80 0.00 0.00 23.06 36.19 4.40 92.00 0.00 0.00 5.00 20.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11.05 98.00 md127 5.00 64.80 0.00 0.00 0.04 12.96 133.80 2014.40 0.00 0.00 0.08 15.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.32 nvme0n1 33.60 822.40 0.00 0.00 0.09 24.48 252.40 3388.10 139.40 35.58 0.04 13.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 7.24 nvme1n1 62.60 3082.40 0.00 0.00 0.10 49.24 427.40 4272.90 177.20 29.31 0.03 10.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 13.70 sda 236.20 10613.60 126.20 34.82 15.19 44.93 11.80 1742.40 18.60 61.18 13.83 147.66 0.00 0.00 0.00 0.00 0.00 0.00 1.40 20.29 3.78 98.14 sdb 35.80 4166.40 56.80 61.34 10.03 116.38 4.20 96.00 0.20 4.55 0.81 22.86 0.00 0.00 0.00 0.00 0.00 0.00 0.80 0.25 0.36 19.04 sdc 118.20 6612.80 83.00 41.25 9.15 55.95 3.00 108.80 0.40 11.76 2.07 36.27 0.00 0.00 0.00 0.00 0.00 0.00 0.60 3.67 1.09 58.20 sdd 207.00 13152.00 166.60 44.59 13.92 63.54 15.80 1100.80 7.80 33.05 6.90 69.67 0.00 0.00 0.00 0.00 0.00 0.00 1.80 9.67 3.01 96.66 sde 55.80 1939.20 20.80 27.15 9.06 34.75 17.40 1885.60 17.80 50.57 7.51 108.37 0.00 0.00 0.00 0.00 0.00 0.00 1.80 1.44 0.64 29.04 sdf 221.40 12720.00 155.20 41.21 14.34 57.45 14.60 1444.80 12.60 46.32 10.51 98.96 0.00 0.00 0.00 0.00 0.00 0.00 1.40 11.71 3.35 97.70 sdg 234.60 14490.40 179.80 43.39 15.66 61.77 9.40 235.20 1.20 11.32 5.36 25.02 0.00 0.00 0.00 0.00 0.00 0.00 2.00 11.10 3.75 97.98 sdh 212.20 9470.40 109.20 33.98 4.05 44.63 19.40 1630.40 11.80 37.82 6.41 84.04 0.00 0.00 0.00 0.00 0.00 0.00 2.00 4.10 0.99 42.78 sdi 214.00 14684.00 186.60 46.58 14.46 68.62 10.00 1197.60 15.20 60.32 11.14 119.76 0.00 0.00 0.00 0.00 0.00 0.00 2.40 13.67 3.24 91.28 sdj 250.60 17296.80 227.00 47.53 18.76 69.02 3.60 92.00 0.80 18.18 5.72 25.56 0.00 0.00 0.00 0.00 0.00 0.00 0.80 16.25 4.73 98.00 `osd_max_scrubs` creates more reads per second (~120 -> ~220), but does not proportionately increase read throughput (as is expected on a spinning disk). Overall, this looks like scrub operations are seek-bound. Looking at `strace -fyp 640105 -e io_submit` output, for lines that involve my HDD `/dev/dm-12`: io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd55baa3000, iov_len=4096}], aio_offset=1654046691328}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f378000, iov_len=24576}], aio_offset=1654046666752}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f262000, iov_len=4096}], aio_offset=1934563819520}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f28b000, iov_len=4096}], aio_offset=1934563823616}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3c7000, iov_len=28672}], aio_offset=2871307956224}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f2d2000, iov_len=4096}], aio_offset=2871307984896}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f34a000, iov_len=8192}], aio_offset=4056494669824}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f35f000, iov_len=4096}], aio_offset=4056494678016}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3e3000, iov_len=53248}], aio_offset=1233895051264}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f0c6000, iov_len=4096}], aio_offset=1233895104512}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f1e2000, iov_len=4096}], aio_offset=2155792314368}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f303000, iov_len=24576}], aio_offset=2155792289792}]) = 2 This looks like it's reading the HDD all over the place, with small reads (`aio_offset`, `iov_len`). Does Ceph scrubbing do a HDD seek for every object? I had imagined that scrubbing might be able to read linearly through the disk (perhaps skipping larger gaps). Is that wrong? Thanks!

7 16

Enable LUKS encryption on a snapshot created from unencrypted image
by Will Gorman 30 Apr '23

30 Apr '23

Is there a way to enable the LUKS encryption format on a snapshot that was created from an unencrypted image without losing data? I've seen in https://docs.ceph.com/en/quincy/rbd/rbd-encryption/ that "Any data written to the image prior to its format may become unreadable, though it may still occupy storage resources." and observed that to be the case when running `encryption format` on an image that already has data in it. However is there any way to take a snapshot of an unencrypted image and enable encryption on the snapshot (or even on a new image cloned from the snapshot?)

2 1

cephfs - max snapshot limit?
by Tobias Hachmer 29 Apr '23

29 Apr '23

Hello, we are running a 3-node ceph cluster with version 17.2.6. For CephFS snapshots we have configured the following snap schedule with retention: /PATH 2h 72h15d6m But we observed that max 50 snapshot are preserved. If a new snapshot is created the oldest 51st is deleted. Is there a limit for maximum cephfs snapshots or maybe this is a bug? I have found the setting "mds_max_snaps_per_dir" which is 100 by default but I think this is not related to my problem? Thanks, Tobias

4 9

Return code -116 when starting MDS scrub
by Thomas Widhalm 29 Apr '23

29 Apr '23

Hi, I followed the steps to repair journal and MDS I found here in the list. I hit a bug that stopped my MDS to start so I took the long way with reading the data. Everything went fine and I can even mount one of my CephFS now. That's a big relieve. But when I start scrub, I just get return code -116 and no scrub is initiatet. I didn't find that code in the docs. Can you help me? [ceph: root@ceph06 /]# ceph tell mds.mds01.ceph06.huavsw scrub start recursive 2023-04-29T10:46:36.926+0000 7ff676ff5700 0 client.79389355 ms_handle_reset on v2:192.168.23.66:6800/1133836262 2023-04-29T10:46:36.953+0000 7ff676ff5700 0 client.79389361 ms_handle_reset on v2:192.168.23.66:6800/1133836262 { "return_code": -116 } (I get the same error, no matter what kind of scrub I start) -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

2 2

import OSD after host OS reinstallation
by Tony Liu 29 Apr '23

29 Apr '23

Hi, The cluster is with Pacific and deployed by cephadm on container. The case is to import OSDs after host OS reinstallation. All OSDs are SSD who has DB/WAL and data together. Did some research, but not able to find a working solution. Wondering if anyone has experiences in this? What needs to be done before host OS reinstallation and what's after? Thanks! Tony

2 5

Block RGW request using Lua
by ondrej＠kuuk.la 28 Apr '23

28 Apr '23

Hello everyone, I've started playing with Lua scripting and would like to ask If anyone knows about a way to drop or close user request on the prerequest context. I would like to block creating buckets with dots in the name, but the use-case could be blocking certain operations, etc. I was able to come up with some like this if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = "InvalidBucketName" Request.Response.Message = "Dots are not allowed." end This works fine, but the bucket is created which is something that I don't want to do. As a dirty workaround, I've thought about changing the bucket name here to an already existing bucket, but the Request.Bucket.Name = "taken" doesn't seem to work as the log gives me an error "attempt to index a nil value (field 'Bucket')". Any help is much appreciated.

1 0

For suggestions and best practices on expanding Ceph cluster and removing old nodes
by huxiaoyu＠horebdata.cn 28 Apr '23

28 Apr '23

Dear Ceph folks, I would like to listen to your advice on the following topic: We have a 6-node Ceph cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes. Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes on Nautilus version。 I can think of two ways to achieve the above goal. The first method would be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes, remove the 6 old nodes from the cluster The second method would get rid of the procedure to upgrade the old 6-node from Luminous to Nautilus, because those 6 nodes will be phased out anyway, but then we have to deal with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and after re-balancing, we can remove the 6 old nodes from the cluster. Any suggestions, advice, or best practice would be highly appreciated. best regards, Samuel huxiaoyu(a)horebdata.cn

7 8

How to call cephfs-top
by E Taka 28 Apr '23

28 Apr '23

I'm using a dockerized Ceph 17.2.6 under Ubuntu 22.04. Presumably I'm missing a very basic thing, since this seems a very simple question: how can I call cephfs-top in my environment? It is not inckuded in the Docker Image which is accessed by "cephadm shell". And calling the version found in the source code always fails with "[errno 13] RADOS permission denied", even when using "--cluster" with the correct ID, "--conffile" and "--id". The auth user client.fstop exists, and "ceph fs perf stats" runs. What am I missing? Thanks!

2 1

Ceph stretch mode / POOL_BACKFILLFULL
by Kilian Ries 28 Apr '23

28 Apr '23

Hi, we run a ceph cluster in stretch mode with one pool. We know about this bug: https://tracker.ceph.com/issues/56650 https://github.com/ceph/ceph/pull/47189 Can anyone tell me what happens when a pool gets to 100% full? At the moment raw OSD usage is about 54% but ceph throws me a "POOL_BACKFILLFULL" error: $ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 63 TiB 29 TiB 34 TiB 34 TiB 54.19 TOTAL 63 TiB 29 TiB 34 TiB 34 TiB 54.19 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 415 MiB 105 1.2 GiB 0.04 1.1 TiB vm_stretch_live 2 64 15 TiB 4.02M 34 TiB 95.53 406 GiB So the pool warning / calculation is just a bug, because it thinks its 50% of the total size. I know ceph will stop IO / set OSDs to read only if the hit a "backfillfull_ratio" ... but what will happen if the pool gets to 100% full ? Will IO still be possible? No limits / quotas are set on the pool ... Thanks Regards, Kilian

2 2

Ceph 16.2.12, particular OSD shows higher latency than others
by Zakhar Kirpichenko 28 Apr '23

28 Apr '23

Hi, I have a Ceph 16.2.12 cluster with uniform hardware, same drive make/model, etc. A particular OSD is showing higher latency than usual in `ceph osd perf`, usually mid to high tens of milliseconds while other OSDs show low single digits, although its drive's I/O stats don't look different from those of other drives. The workload is mainly random 4K reads and writes, the cluster is being used as Openstack VM storage. Is there a way to trace, which particular PG, pool and disk image or object cause this OSD's excessive latency? Is there a way to tell Ceph to I would appreciate any advice or pointers. Best regards, Zakhar

3 9

Massive OMAP remediation
by Ben.Zieglmeier 27 Apr '23

27 Apr '23

Hi All, We have a RGW cluster running Luminous (12.2.11) that has one object with an extremely large OMAP database in the index pool. Listomapkeys on the object returned 390 Million keys to start. Through bilog trim commands, we’ve whittled that down to about 360 Million. This is a bucket index for a regrettably unsharded bucket. There are only about 37K objects actually in the bucket, but through years of neglect, the bilog grown completely out of control. We’ve hit some major problems trying to deal with this particular OMAP object. We just crashed 4 OSDs when a bilog trim caused enough churn to knock one of the OSDs housing this PG out of the cluster temporarily. The OSD disks are 6.4TB NVMe, but are split into 4 partitions, each housing their own OSD daemon (collocated journal). We want to be rid of this large OMAP object, but are running out of options to deal with it. Reshard outright does not seem like a viable option, as we believe the deletion would deadlock OSDs can could cause impact. Continuing to run `bilog trim` 1000 records at a time has been what we’ve done, but this also seems to be creating impacts to performance/stability. We are seeking options to remove this problematic object without creating additional problems. It is quite likely this bucket is abandoned, so we could remove the data, but I fear even the deletion of such a large OMAP could bring OSDs down and cause potential for metadata loss (the other bucket indexes on that same PG). Any insight available would be highly appreciated. Thanks.

3 3

Bucket notification
by Szabo, Istvan (Agoda) 27 Apr '23

27 Apr '23

Hi, I'm trying to set a kafka endpoint for bucket object create operation notifications but the notification is not created in kafka endpoint. Settings seems to be fine because I can upload to the bucket objects when these settings are applied: NotificationConfiguration> <TopicConfiguration> <Id>bulknotif</Id> <Topic>arn:aws:sns:default::butcen</Topic> <Event>s3:ObjectCreated:*</Event> <Event>s3:ObjectRemoved:*</Event> </TopicConfiguration> </NotificationConfiguration> but it simply not created any message in kafka. This is my topic creation post request: https://xxx.local/? Action=CreateTopic& Name=butcen& kafka-ack-level=broker& use-ssl=true& push-endpoint=kafka://ceph:pw@xxx.local:9093 Am I missing something or it's definitely kafka issue? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Bucket empty after resharding on multisite environment
by Boris Behrens 27 Apr '23

27 Apr '23

Hi, I just resharded a bucket on an octopus multisite environment from 11 to 101. I did it on the master zone and it went through very fast. But now the index is empty. The files are still there when doing a radosgw-admin bucket radoslist --bucket-id Do I just need to wait or do I need to recover that somehow?

1 2

MDS recovery
by jack＠translucencebio.com 27 Apr '23

27 Apr '23

Hi All, We have a CephFS cluster running Octopus with three control nodes each running an MDS, Monitor, and Manager on Ubuntu 20.04. The OS drive on one of these nodes failed recently and we had to do a fresh install, but made the mistake of installing Ubuntu 22.04 where Octopus is not available. We tried to force apt to use the Ubuntu 20.04 repo when installing Ceph so that it would install Octopus, but for some reason Quincy was still installed. We re-integrated this node and it seemed to work fine for about a week until our cluster reported damage to an MDS daemon and placed our filesystem into a degraded state. cluster: id: 692905c0-f271-4cd8-9e43-1c32ef8abd13 health: HEALTH_ERR mons are allowing insecure global_id reclaim 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged noout flag(s) set 161 scrub errors Possible data damage: 24 pgs inconsistent 8 pgs not deep-scrubbed in time 4 pgs not scrubbed in time 6 daemons have recently crashed services: mon: 3 daemons, quorum database-0,file-server,webhost (age 12d) mgr: database-0(active, since 4w), standbys: webhost, file-server mds: cephfs:0/1 3 up:standby, 1 damaged osd: 91 osds: 90 up (since 32h), 90 in (since 5M) flags noout task status: data: pools: 7 pools, 633 pgs objects: 169.18M objects, 640 TiB usage: 883 TiB used, 251 TiB / 1.1 PiB avail pgs: 605 active+clean 23 active+clean+inconsistent 4 active+clean+scrubbing+deep 1 active+clean+scrubbing+deep+inconsistent We are not sure if the Quincy/Octopus version mismatch is the problem, but we are in the process of downgrading this node now to ensure all nodes are running Octopus. Before doing that, we ran the following commands to try and recover: $ cephfs-journal-tool --rank=cephfs:all journal export backup.bin $ sudo cephfs-journal-tool --rank=cephfs:all event recover_dentries summary: Events by type: OPEN: 29589 PURGED: 1 SESSION: 16 SESSIONS: 4 SUBTREEMAP: 127 UPDATE: 70438 Errors: 0 $ cephfs-journal-tool --rank=cephfs:0 journal reset: old journal was 170234219175~232148677 new journal start will be 170469097472 (2729620 bytes past old end) writing journal head writing EResetJournal entry done $ cephfs-table-tool all reset session All of our MDS daemons are down and fail to restart with the following errors: -3> 2023-04-20T10:25:15.072-0700 7f0465069700 -1 log_channel(cluster) log [ERR] : journal replay alloc 0x1000053af79 not in free [0x1000053af7d~0x1e8,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2,0x1000053b561~0x2,0x1000053b565~0x1de,0x1000053b938~0x1fd,0x1000053bd2a~0x4,0x1000053bf23~0x4,0x1000053c11c~0x4,0x1000053cd7b~0x158,0x1000053ced8~0xffffac3128] -2> 2023-04-20T10:25:15.072-0700 7f0465069700 -1 log_channel(cluster) log [ERR] : journal replay alloc [0x1000053af7a~0x1eb,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2], only [0x1000053af7d~0x1e8,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2] is in free [0x1000053af7d~0x1e8,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2,0x1000053b561~0x2,0x1000053b565~0x1de,0x1000053b938~0x1fd,0x1000053bd2a~0x4,0x1000053bf23~0x4,0x1000053c11c~0x4,0x1000053cd7b~0x158,0x1000053ced8~0xffffac3128] -1> 2023-04-20T10:25:15.072-0700 7f0465069700 -1 /build/ceph-15.2.15/src/mds/journal.cc: In function 'void EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)' thread 7f0465069700 time 2023-04-20T10:25:15.076784-0700 /build/ceph-15.2.15/src/mds/journal.cc: 1513: FAILED ceph_assert(inotablev == mds->inotable->get_version()) ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x155) [0x7f04717a3be1] 2: (()+0x26ade9) [0x7f04717a3de9] 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x67e2) [0x560feaca36f2] 4: (EUpdate::replay(MDSRank*)+0x42) [0x560feaca5bd2] 5: (MDLog::_replay_thread()+0x90c) [0x560feac393ac] 6: (MDLog::ReplayThread::entry()+0x11) [0x560fea920821] 7: (()+0x8609) [0x7f0471318609] 8: (clone()+0x43) [0x7f0470ee9163] 0> 2023-04-20T10:25:15.076-0700 7f0465069700 -1 *** Caught signal (Aborted) ** in thread 7f0465069700 thread_name:md_log_replay ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable) 1: (()+0x143c0) [0x7f04713243c0] 2: (gsignal()+0xcb) [0x7f0470e0d03b] 3: (abort()+0x12b) [0x7f0470dec859] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f04717a3c3c] 5: (()+0x26ade9) [0x7f04717a3de9] 6: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x67e2) [0x560feaca36f2] 7: (EUpdate::replay(MDSRank*)+0x42) [0x560feaca5bd2] 8: (MDLog::_replay_thread()+0x90c) [0x560feac393ac] 9: (MDLog::ReplayThread::entry()+0x11) [0x560fea920821] 10: (()+0x8609) [0x7f0471318609] 11: (clone()+0x43) [0x7f0470ee9163] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. At this point, we decided it's best to ask for some guidance before issuing any other recovery commands. Can anyone advise what we should do?

2 1

How to find the bucket name from Radosgw log?
by viplanghe6＠gmail.com 27 Apr '23

27 Apr '23

I find a log like this, and I thought the bucket name should be "photos": ....[2023-04-19 15:48:47.0.5541s] "GET /photos/shares/.... But I can not find it: radosgw-admin bucket stats --bucket photos failure: 2023-04-19 15:48:53.969 7f69dce49a80 0 could not get bucket info for bucket=photos (2002) Unknown error 2002 How does this happen? Thanks

3 2

Re: Disks are filling up
by Omar Siam 27 Apr '23

27 Apr '23

No this is a cephadm setup. Not rook. In the last days it is still deep scrubbing and filling up. We have to do something about it as it now impacts our K8s cluster (very slow cephfs access) and we are running out of (allocated) diskspace again. Some more details now that I had a few more days to think about our particular setup: * This is a setup with ESXi/vSphere virtualization. The ceph nodes are just some VMs. We don't have access to the bare servers or even direct access to the HDDs/SSDs ceph runs on. * The setup is "asymmetric": there are 2 nodes on SSDs and one on HDDs (they are all RAIDx with hardware controllers, but we have no say in this). I labeled all OSDs as HDDs (even when VMWare reported SSD). * We looked at the OSDs device usage and it is 100% (from the VMs point of view) for the HDDs (20% on everage for the SSD nodes). My suspision is: * deep scrubbing means every new write goes to unallocated space, no more overwrite/deleting while deep scrubbing. I didn't find it in the docs. Maybe I missed it, maybe that is common wisdom among the initiated. * We write more new data per second to cephfs than can be scrubbed so scrubbing never ends and the PGs fill up. We now ordered SSDs for the HDD only node to prevent this in the future. Meanwhile we need to do something so we think about moving the data in cephfs to a new PG that does not need deep scrubbing at the moment. Also we think about moving the OSD from the physical host that only has HDDs to one with SSDs ruining redundancy for a short while and hoping for the best Am 26.04.2023 um 02:28 schrieb A Asraoui: > > Omar, glad to see cephfs with kubernetes up and running.. did you guys > use rook to deploy this ?? > > Abdelillah > On Mon, Apr 24, 2023 at 6:56 AM Omar Siam <Omar.Siam(a)oeaw.ac.at> wrote: > > Hi list, > > we created a cluster for using cephfs with a kubernetes cluster. > Since a > few weeks now the cluster keeps filling up at an alarming rate > (100 GB per day). > This is while the most relevant pg is deep scrubbing and was > interupted > a few times. > > We use about 150G (du using the mounted filesystem) on the cephfs > filesystem and try not to use snapshots (.snap directories "exist" > but > are empty). > We do not understand why the pgs get bigger and bigger while cephfs > stays about the same size (overwrites on files certainly happen). > I suspect some snapshots mechanism. Any ideas how to debug this to > stop it? > > Maybe we should try to speed up the deep scrubbing somehow? > Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at |www.oeaw.ac.at/acdh

1 0

Veeam backups to radosgw seem to be very slow
by Boris Behrens 27 Apr '23

27 Apr '23

We have a customer that tries to use veeam with our rgw objectstorage and it seems to be blazingly slow. What also seems to be strange, that veeam sometimes show "bucket does not exist" or "permission denied". I've tested parallel and everything seems to work fine from the s3cmd/aws cli standpoint. Does anyone here ever experienced veeam problems with rgw? Cheers Boris

5 5

Rados gateway data-pool replacement.
by Gaël THEROND 26 Apr '23

26 Apr '23

Hi everyone, quick question regarding radosgw zone data-pool. I’m currently planning to migrate an old data-pool that was created with inappropriate failure-domain to a newly created pool with appropriate failure-domain. If I’m doing something like: radosgw-admin zone modify —rgw-zone default —data-pool <new_pool> Will data from the old pool be migrated to the new one or do I need to do something else to migrate those data out of the old pool? I’ve read a lot of mail archive with peoples willing to do that but I can’t get a clear answer from those archives. I’m running on nautilus release of it ever help. Thanks a lot! PS: This mail is a redo of the old one as I’m not sure the former one worked (missing tags).

3 4

Ceph 16.2.12, bluestore cache doesn't seem to be used much
by Zakhar Kirpichenko 26 Apr '23

26 Apr '23

Hi, I have a Ceph 16.2.12 cluster with hybrid OSDs (HDD block storage, DB/WAL on NVME). All OSD settings are default except, cache-related settings are as follows: osd.14 dev bluestore_cache_autotune true osd.14 dev bluestore_cache_size_hdd 4294967296 osd.14 dev bluestore_cache_size_ssd 4294967296 osd.14 advanced bluestore_default_buffered_write false osd.14 dev osd_memory_cache_min 2147483648 osd.14 basic osd_memory_target 17179869184 Other settings such as bluestore_cache_kv_ratio, bluestore_cache_meta_ratio, etc. are default. I.e. OSD memory target is set to 16 GB, bluestore cache is set to 4 GB for HDDs and SSDs, minimum cache size is 2 GB. When I dump memory pools of OSDs, bluestore cache doesn't seem to be actively used (https://pastebin.com/EpfFp85C) despite there's plenty of memory and the memory target is 16 GB, memory pools are around 2 GB and the total RSS of the OSD process is ~4.8 GB. There are 66 OSDs in the cluster and the situation is very similar with all of them. The OSDs are being used quite actively for both reads and writes, and I guess they could benefit from using more memory for caching, especially considering that we have lots of RAM available on each host. Is there a way to increase and/or tune OSD cache memory usage? I would appreciate any advice or pointers. Best regards, Zakhar

1 0

Object data missing, but metadata is OK (Quincy 17.2.3)
by Jeff Briden 26 Apr '23

26 Apr '23

We have a large cluster on Quincy 17.2.3 with a bucket holding 8.9 million small (15~20 MiB) objects. All the objects were multipart uploads from scripts using `aws s3 cp` The data is static (write-once, read-many) with no manual deletions and no new writes for months. We recently found 3 objects in this bucket that cannot be retrieved. The symptom is exactly the same as https://tracker.ceph.com/issues/47866 and https://bugzilla.redhat.com/show_bug.cgi?id=1892644 which were fixed a long time ago. Any form of listing (`aws s3 ls`, radosgw-admin object stat, radoslist, http head request, etc) returns good data, but the objects cannot be retrieved and rados -p ls shows the object data is missing. Any suggestions on how to troubleshoot this further?

1 0

How to control omap capacity？
by WeiGuo Ren 26 Apr '23

26 Apr '23

I have two osds. these osd are used to rgw index pool. After a lot of stress tests, these two osds were written to 99.90%. The full ratio (95%) did not take effect? I don't know much. Could it be that if the osd of omap is fully stored, it cannot be limited by the full ratio? ALSO I use ceph-bluestore-tool to expand it . Before I add a partition . But i failed, I dont know why. In my cluster every osd have 55GB (db val data in same device), ceph -v is 14.2.5. can anyone give me some idear to fix it?

2 1

Bug, pg_upmap_primaries.empty()
by Nguetchouang Ngongang Kevin 26 Apr '23

26 Apr '23

Good morning, i found a bug on ceph reef After installing ceph and deploying 9 osds with a cephfs layer. I got this error after many writing and reading operations on the ceph fs i deployed. ```{ "assert_condition": "pg_upmap_primaries.empty()", "assert_file": "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3593-g1e73409b/rpm/el8/BUILD/ceph-18.0.0-3593-g1e73409b/src/osd/OSDMap.cc", "assert_func": "void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const", "assert_line": 3239, "assert_msg": "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3593-g1e73409b/rpm/el8/BUILD/ceph-18.0.0-3593-g1e73409b/src/osd/OSDMap.cc: In function 'void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const' thread 7f86cb8e5700 time 2023-04-26T12:25:12.278025+0000\n/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3593-g1e73409b/rpm/el8/BUILD/ceph-18.0.0-3593-g1e73409b/src/osd/OSDMap.cc: 3239: FAILED ceph_assert(pg_upmap_primaries.empty())\n", "assert_thread_name": "msgr-worker-0", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7f86d0d21cf0]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x55ce1794774b]", "/usr/bin/ceph-osd(+0x6368b7) [0x55ce179478b7]", "(OSDMap::encode(ceph::buffer::v15_2_0::list&, unsigned long) const+0x1229) [0x55ce183e0449]", "(MOSDMap::encode_payload(unsigned long)+0x396) [0x55ce17ae2576]", "(Message::encode(unsigned long, int, bool)+0x2e) [0x55ce1825dbee]", "(ProtocolV1::prepare_send_message(unsigned long, Message*, ceph::buffer::v15_2_0::list&)+0x54) [0x55ce184e5914]", "(ProtocolV1::write_event()+0x511) [0x55ce184f4ce1]", "(EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa64) [0x55ce182eb484]", "/usr/bin/ceph-osd(+0xfdf276) [0x55ce182f0276]", "/lib64/libstdc++.so.6(+0xc2b13) [0x7f86d0369b13]", "/lib64/libpthread.so.0(+0x81ca) [0x7f86d0d171ca]", "clone()" ], "ceph_version": "18.0.0-3593-g1e73409b", "crash_id": "2023-04-26T12:25:12.286947Z_55675d7c-7833-4e91-b0eb-6df705104c2e", "entity_name": "osd.0", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "0ffad2c4bc07caf68ff1e124d3911823bc6fa6f5772444754b7f0a998774c8fe", "timestamp": "2023-04-26T12:25:12.286947Z", "utsname_hostname": "node1-link-1", "utsname_machine": "x86_64", "utsname_release": "5.4.0-100-generic", "utsname_sysname": "Linux", "utsname_version": "#113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022" } ``` I really don't know what is this error for, Will appreciate any help. Cordially, -- Nguetchouang Ngongang Kevin ENS de Lyon https://perso.ens-lyon.fr/kevin.nguetchouang/

2 2

Ceph Leadership Team meeting minutes - 2023 April 26
by Casey Bodley 26 Apr '23

26 Apr '23

# ceph windows tests PR check will be made required once regressions are fixed windows build currently depends on gcc11 which limits use of c++20 features. investigating newer gcc or clang toolchain # 16.2.13 release final testing in progress # prometheus metric regressions https://tracker.ceph.com/issues/59505 related to previous discussion on 4/12 about quincy backports integration test coverage needed for ceph-exporter and the mgr module # lab update centos/rhel tests were failing due to problematic mirrorlists fixed in https://github.com/ceph/ceph-cm-ansible/pull/731 more sanity checks in progress at https://github.com/ceph/ceph-cm-ansible/pull/733 # cephalocon feedback dev summit etherpads: https://pad.ceph.com/p/cephalocon-dev-summit-2023 collect more notes here: https://pad.ceph.com/p/cephalocon-2023-brainstorm request for dev-focused longer term discussion could have specific user-focused and dev-focused sessions dense conference, hard to fit everything in 3 days could have longer component updates during conf, with time for questions perhaps 3 days of conf, dev-specific discussions a day before (no cfp, one big room, then option for breakout), user-feedback sessions during the normal con

1 0

Re: Move ceph to new addresses and hostnames
by Jan Marek 26 Apr '23

26 Apr '23

Hello all, today I moved ceph to HEALTH_OK state :-) 1) I had to restart MGR node, then my old c-osdx hostnames goes definitely away and all of OSDs from old machines are now orchestrated by 'ceph orch' command. 2) I've updated ceph* packages on the osd2 node to version 17.2.6, then I tried 'cephadm adopt' command once more and voila! It works like a charm. I will try to configure OSDs on the node 1 to adopt WAL and DB from prepared LVM... Maybe after upgrade to newer version of CEPH it will be OK? Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 1

OSD_TOO_MANY_REPAIRS on random OSDs causing clients to hang
by Thomas Hukkelberg 26 Apr '23

26 Apr '23

Hi all, Over the last 2 weeks we have experienced several OSD_TOO_MANY_REPAIRS errors that we struggle to handle in a non-intrusive manner. Restarting MDS + hypervisor that accessed the object in question seems to be the only way we can clear the error so we can repair the PG and recover access. Any pointers on how to handle this issue in a more gentle way than rebooting the hypervisor and failing the MDS would be welcome! The problem seems to only affect one specific pool (id 42) that is used for cephfs_data. This pool is our second cephfs data pool in this cluster. The data in the pool is accessible via LXC container via Samba and have the cephfs filesystem bind-mounted from hypervisor. Ceph is recently updated to version 16.2.11 (pacific) -- kernel version is 5.13.19-6-pve on OSD-hosts/samba-containers and 5.19.17-2-pve on MDS-hosts. The following warnings are issued: $ ceph health detail HEALTH_WARN 1 clients failing to respond to capability release; Too many repaired reads on 1 OSDs; Degraded data redundancy: 1/2648430 090 objects degraded (0.000%), 1 pg degraded; 1 slow ops, oldest one blocked for 608 sec, osd.34 has slow ops [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.hk-cephnode-65(mds.0): Client hk-cephnode-56 failing to respond to capability release client_id: 9534859837 [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs osd.34 had 9936 reads repaired [WRN] PG_DEGRADED: Degraded data redundancy: 1/2648430090 objects degraded (0.000%), 1 pg degraded pg 42.e2 is active+recovering+degraded+repair, acting [34,275,284] [WRN] SLOW_OPS: 1 slow ops, oldest one blocked for 608 sec, osd.34 has slow ops The logs for OSD.34 are flooded with these messages: root@hk-cephnode-53:~# tail /var/log/ceph/ceph-osd.34.log 2023-04-26T11:41:00.760+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 missing primary copy of 42:4703efac:::10003d86a99.00000001:head, will try copies on 275,284 2023-04-26T11:41:00.784+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.812+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.812+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 missing primary copy of 42:4703efac:::10003d86a99.00000001:head, will try copies on 275,284 2023-04-26T11:41:00.824+0200 7f03a821f700 -1 osd.34 1352563 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.9534859837.0:20412906 42.e2 42:4703efac:::10003d86a99.00000001:head [read 0~1048576 [307@0] out=1048576b] snapc 0=[] RETRY=5 ondisk+retry+read+known_if_redirected e1352553) 2023-04-26T11:41:00.824+0200 7f03a821f700 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'qa-cephfs_data' : 1 ]) 2023-04-26T11:41:00.840+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.864+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.864+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 missing primary copy of 42:4703efac:::10003d86a99.00000001:head, will try copies on 275,284 2023-04-26T11:41:00.888+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head We have tried the following: - Restarting the OSD in question clears the error for a few seconds but then we also we get OSD_TOO_MANY_REPAIRS on OSDs with PGs that holds the object that have blocked I/O. - Trying to repair the PG seems to restart every 10 second and not actually do anything/progressing. (Is there a way to check repair progress?) - Restarting the MDS and hypervisor clears the error (the hypervisor hangs for several minutes before timing out). However if the object is requested again the error reoccurs. If we don't access the object we are able to eventually repair the PG. - Occasionally setting the primary-affinity to 0 for the primary OSD in the PG clears the error after restarting all affected OSD and we are able to repair the PG (unless the object is accessed during recovery) and access to the object is OK afterwards. - Finding and deleting the file pointing to the object (10003d86a99) and restarting OSDs will clear the error. - Killing the samba process that accessed the object does not clear the SLOW_OPS, and hence the error prevail - Normal scrubs have revealed a handfull of other PGs in the same pool (id 42) that are damaged and we are doing repairs without any problems. - We believe MDS_CLIENT_LATE_RELEASE and SLOW_OPS errors are symptoms of the fact that the I/O are blocked. - We have verified that there are no SMART errors of any kind on any of our disks in the cluster. - If we don't handle this issue rather promptly, we experience full lockup of the samba container and rebooting hypervisor seems to be the only cure. Trying to force unmount and remount cephfs does not help. This have now happened 6-7 times over the last 2 weeks and we suspect that a hardware or memory error on one of our nodes may have caused the objects to be written to disk with bad checksums. We have replaced the mainboard in one of our nodes that we might think is the culprit and are currently testing the memory. Can these random checksum errors be caused by anything else that we should investigate? It's a bit suspicious that the error only occurs on one specific pool? If the mainboard are to blame we should see these errors in more pools by now? Regardless we are stumped by how Ceph handles this error. Checksum-errors should not leave clients hanging like this? Should this be considered a bug? Is there a way to cancel the blocking I/O request to clear the error? And why is the PG flapping between active+recovering+degraded+repair, active+recovering+repair, active+clean+repair every few seconds? Any ideas on how to gracefully battle this problem? Thanks! --thomas Thomas Hukkelberg thomas(a)hovedkvarteret.no

3 3

Increase timeout for marking osd down
by Nicola Mori 26 Apr '23

26 Apr '23

Dear Ceph users, my cluster is made of very old machines on a Gbit ethernet. I see that sometimes some OSDs are marked down due to slow networking, especially on heavy network load like during recovery. This causes problems, for example PGs keeps being deactivated and activated as the OSDs are marked down and up (at least to my best understanding). So I'd need to know if there is some way to increase the timeout after which an OSD is marked down, to cope with my slow network. Thanks, Nicola

2 1

ceph pg stuck - missing on 1 osd how to proceed
by xadhoom76＠gmail.com 26 Apr '23

26 Apr '23

Hi to all Using ceph 17.2.5 i have 3 pgs in stuck state ceph pg map 8.2a6 osdmap e32862 pg 8.2a6 (8.2a6) -> up [88,100,59] acting [59,100] looking at it ho 88 ,100 and 59 i got that ceph pg ls-by-osd osd.100 | grep 8.2a6 8.2a6 211004 209089 0 0 174797925205 0 0 7075 active+undersized+degraded+remapped+backfilling 21m 32862'1540291 32862:3387785 [88,100,59]p88 [59,100]p59 2023-03-12T08:08:00.903727+0000 2023-03-12T08:08:00.903727+0000 6839 queued for deep scrub ceph pg ls-by-osd osd.59 | grep 8.2a6 8.2a6 211005 209084 0 0 174798941087 0 0 7076 active+undersized+degraded+remapped+backfilling 22m 32862'1540292 32862:3387798 [88,100,59]p88 [59,100]p59 2023-03-12T08:08:00.903727+0000 2023-03-12T08:08:00.903727+0000 6839 queued for deep scrub BUT ceph pg ls-by-osd osd.88 | grep 8.2a6 ---> NONE it is missing .... how to proceed ? Best regards

4 7

Dead node (watcher) won't timeout on RBD
by max＠netpunt.nl 26 Apr '23

26 Apr '23

Hey all, I recently had a k8s node failure in my homelab, and even though I powered it off (and it's done for, so it won't get back up), it still shows up as watcher in rbd status. ``` root@node0:~# rbd status kubernetes/csi-vol-3e7af8ae-ceb6-4c94-8435-2f8dc29b313b Watchers: watcher=10.0.0.103:0/1520114202 client.1697844 cookie=140289402510784 watcher=10.0.0.103:0/39967552 client.1805496 cookie=140549449430704 root@node0:~# ceph osd blocklist ls 10.0.0.103:0/0 2023-04-15T13:15:39.061379+0200 listed 1 entries ``` Even though the node is down & I have blocked it multiple times for hours, it won't disappear. Meaning, ceph-csi-rbd claims the image is mounted already (manually binding works fine, and can cleanly unbind as well, but can't unbind from a node that doesn't exist anymore). Is there any possibility to force kick an rbd client / watcher from ceph (e.g. switching the mgr / mon) or to see why this is not timing out? I found some historical mails & issues (related to rook, which I don't use) regarding a param `osd_client_watch_timeout` but can't find how that relates to the RBD images. Cheers, Max.

2 1

Ovirt integration with Ceph
by kushagra.gupta＠hsc.com 26 Apr '23

26 Apr '23

Hi Team, We are trying to integrate ceph with ovirt. We have deployed ovirt 4.4. We want to create a storage domain of POSIX compliant type for mounting a ceph based infrastructure in ovirt. We have done SRV based resolution in our DNS server for ceph mon nodes but we are unable to create a storage domain using that. We are able to manually mount the ceph-mon nodes using the following command on the deployment hosts: sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 -o rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA== [root@deployment-host mnt]# df -kh df: /run/user/0/gvfs: Transport endpoint is not connected Filesystem Size Used Avail Use% Mounted on [abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 19G 0 19G 0% /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 Query: 1. Could anyone help us out with storage domain creation in ovirt for SRV resolved ceph-mon nodes.

2 1

Could you please explain the PG concept
by wodel youchi 26 Apr '23

26 Apr '23

Hi, I am learning Ceph and I am having a hard time understanding PG and PG calculus . I know that a PG is a collection of objects, and that PG are replicated over the hosts to respect the replication size, but... In traditional storage, we use size in Gb, Tb and so on, we create a pool from a bunch of disks or raid arrays of some size then we create volumes of a certain size and use them. If the storage is full we add disks, then we extend our pools/volumes. The idea of size is simple to understand. Ceph, although it supports the notion of pool size in Gb, Tb ...etc. Pools are created using PGs, and now there is also the notion of % of data. When I use pg calc from ceph or from redhat, the generated yml file contains the % variable, but the commands file contains only the PGs, and when you are configuring 15% and 18% have the same number of PGs !!!!!!!!!!!!??? The pg calc encourages you to create a %data multiple of 100, in other words, it assumes that you know all your pools from the start. What if you won't consume all your raw disk space. What happens when you need to add a new pool? Also when you create several pools, and then execute ceph osd df tree, you can see that all pools show the raw size as a free space, it is like all pools share the same raw space regardless of their PG number. If someone can put some light on this concept and how to manage it wisely, because the documentation keeps saying that it's an important concept, that you have to pay attention when choosing the number of PGs for a pool from the start. Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

3 2

Reset a bucket in a zone
by Yixin Jin 25 Apr '23

25 Apr '23

Hi folks, Within a zonegroup, once a bucket is created, its metadata is sync-ed over to all zones. With bucket-level sync policy, however, its content may or may not be sync-ed over. To simplify the sync process, sometime I'd like to pick the bucket in a zone as the absolute truth and sync its content over to the bucket in another zone, which may have done some local deletions since the last time they were sync-ed. I don't want those local deletions to mess up with the planned sync. Is it possible to reset the bucket in this zone so it is in a "pristine" state and will receive everything from the source? Thanks,Yixin

1 0

Rados gateway lua script-package error lib64
by Thomas Bennett 25 Apr '23

25 Apr '23

Hi, I've noticed that when my lua script runs I get the following error on my radosgw container. It looks like the lib64 directory is not included in the path when looking for shared libraries. Copying the content of lib64 into the lib directory solves the issue on the running container. Here are more details: Apr 25 20:26:59 xxx-ceph-xxxx radosgw[60901]: req 2268223694354647302 0.000000000s Lua ERROR: /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*share*/lua/5.3/socket.lua:12: module 'socket.core' not found: no field package.preload['socket.core'] no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*share* /lua/5.3/socket/core.lua' no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*lib* /lua/5.3/socket/core.so' no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*lib* /lua/5.3/socket.so' As mentioned the following command on the running radosgw container solves the issue for the running container: cp -ru /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib64/* /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib/ Cheers, Tom

1 0

cephadm grafana per host certificate
by Eugen Block 25 Apr '23

25 Apr '23

Hi *, I've set up grafana, prometheus and node-exporter on an adopted cluster (currently running 16.2.10) and was trying to enable ssl for grafana. As stated in the docs [1] there's a way to configure individual certs and keys per host: ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i $PWD/certificate.pem So I did that, then ran 'ceph orch reconfig grafana' but I still get a bad cert error message: Apr 20 10:21:19 ceph01 conmon[3772491]: server.go:3160: http: TLS handshake error from <IP>:46084: remote error: tls: bad certificate It seems like the cephadm generated cert/key pair (mgr/cephadm/grafana_key; mgr/cephadm/grafana_crt) supersedes the per-host certs, and even after removing the generated cert/key (and then reconfigure) cephadm regenerates a them and leaves me with the same problem. Is this a known issue and what would be the fix? I didn't find anything on tracker, but I might have missed it. To confirm that my custom certs actually work I replaced the general cert with my custom cert and the error doesn't appear, I can see the grafana graphs in the dashboard. I could leave it like this, but if grafana would failover it wouldn't work anymore, of course. Any hints are greatly appreciated. Thanks, Eugen [1] https://docs.ceph.com/en/latest/cephadm/services/monitoring/#configuring-ss…

2 4

Move ceph to new addresses and hostnames
by Jan Marek 25 Apr '23

25 Apr '23

Hi there, I have ceph cluster created by ceph-volume - bluestore, in every node is 12 HDD and 1 NVMe, which is divided to 24 LVM partition for DB and WAL. I've turned this cluster to 'ceph orch' management, then I've moved to quincy release (now I'm using a 17.2.5 version). I had to move whole cluster to another addreses and another hostnames. MON, MGR and MDS goes without problem, but OSD was really pain process :-( Now I have cluster with this problem: # ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.mon1.ulytsa mon1 running (11w) 3m ago 11w 5609M - 17.2.5 cc65afd6173a db1aa336263a mds.cephfs.mon2.zxhxqk mon2 running (11w) 3m ago 11w 33.1M - 17.2.5 cc65afd6173a 5b9ced4a4b71 mds.cephfs.mon3.rpkvlt mon3 running (11w) 3m ago 11w 32.4M - 17.2.5 cc65afd6173a 045e23f124aa mgr.mon1.buqyga mon1 *:8080 running (11w) 3m ago 11w 2300M - 17.2.5 cc65afd6173a 9577239527b5 mgr.mon2.goghws mon2 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 4fb1ae26765e mgr.mon3.slpgay mon3 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 06e491084a5e mon.mon1 mon1 running (11w) 3m ago 11w 1576M 2048M 17.2.5 cc65afd6173a 2f18c737faa9 mon.mon2 mon2 running (11w) 3m ago 11w 1598M 2048M 17.2.5 cc65afd6173a 31091cbbfb8e mon.mon3 mon3 running (11w) 3m ago 11w 1463M 2048M 17.2.5 cc65afd6173a 4d0b094c9ca1 osd.0 osd1 running (9w) 3m ago 10w 5133M 3745M 17.2.5 cc65afd6173a 3b28e48d3630 osd.1 osd1 running (7w) 3m ago 10w 5425M 3745M 17.2.5 cc65afd6173a 3336ccdfd232 osd.2 osd1 running (9w) 3m ago 10w 5223M 3745M 17.2.5 cc65afd6173a e8fc077aef59 osd.3 osd1 running (9w) 3m ago 10w 5050M 3745M 17.2.5 cc65afd6173a 4fbf34450237 osd.4 osd1 running (9w) 3m ago 10w 7526M 3745M 17.2.5 cc65afd6173a a4875c354540 osd.5 osd1 running (9w) 3m ago 10w 4854M 3745M 17.2.5 cc65afd6173a b006526228ae osd.6 osd1 running (9w) 3m ago 10w 6498M 3745M 17.2.5 cc65afd6173a 4c326271e188 osd.7 osd1 running (9w) 3m ago 10w 4410M 3745M 17.2.5 cc65afd6173a ca0f3ce31031 osd.8 osd1 running (9w) 3m ago 10w 7337M 3745M 17.2.5 cc65afd6173a 99269a832819 osd.9 osd1 running (9w) 3m ago 10w 4717M 3745M 17.2.5 cc65afd6173a f39ce0bb5316 osd.10 osd1 running (9w) 3m ago 10w 4295M 3745M 17.2.5 cc65afd6173a 0871793fa261 osd.11 osd1 running (9w) 3m ago 10w 5552M 3745M 17.2.5 cc65afd6173a 32a8b589b3bd osd.24 osd3 running (109m) 3m ago 6M 3306M 3745M 17.2.5 cc65afd6173a 466d80a55d96 osd.25 osd3 running (109m) 3m ago 6M 3145M 3745M 17.2.5 cc65afd6173a b1705621116a osd.26 osd3 running (109m) 3m ago 6M 3063M 3745M 17.2.5 cc65afd6173a c30253a1a83f osd.27 osd3 running (109m) 3m ago 6M 3257M 3745M 17.2.5 cc65afd6173a aa0a647d93f1 osd.28 osd3 running (109m) 3m ago 6M 2244M 3745M 17.2.5 cc65afd6173a d3c68ed6572b osd.29 osd3 running (109m) 3m ago 6M 3509M 3745M 17.2.5 cc65afd6173a 2c425b17abf7 osd.30 osd3 running (109m) 3m ago 6M 3814M 3745M 17.2.5 cc65afd6173a 44747256b34a osd.31 osd3 running (109m) 3m ago 6M 2958M 3745M 17.2.5 cc65afd6173a b7b7946fa24e osd.32 osd3 running (109m) 3m ago 6M 3016M 3745M 17.2.5 cc65afd6173a fc9c024fed4f osd.33 osd3 running (109m) 3m ago 6M 5366M 3745M 17.2.5 cc65afd6173a edc2dbd9c556 osd.34 osd3 running (109m) 3m ago 6M 4577M 3745M 17.2.5 cc65afd6173a 46d7668742cf osd.35 osd3 running (109m) 3m ago 6M 2538M 3745M 17.2.5 cc65afd6173a 96a15a9ad3d7 osd.36 osd4 running (103m) 3m ago 8w 2707M 3745M 17.2.5 cc65afd6173a adf884af609b osd.37 osd4 running (103m) 3m ago 6M 3347M 3745M 17.2.5 cc65afd6173a 8f824026c6ae osd.38 osd4 running (103m) 3m ago 6M 3377M 3745M 17.2.5 cc65afd6173a 2a70c0b860ff osd.39 osd4 running (103m) 3m ago 6M 2814M 3745M 17.2.5 cc65afd6173a 4d5833f1faaf osd.40 osd4 running (103m) 3m ago 6M 2633M 3745M 17.2.5 cc65afd6173a e2e492c4f4a6 osd.41 osd4 running (103m) 3m ago 6M 2141M 3745M 17.2.5 cc65afd6173a 80f67faf1238 osd.42 osd4 running (103m) 3m ago 6M 1527M 3745M 17.2.5 cc65afd6173a c1aef5891ad5 osd.43 osd4 running (103m) 3m ago 6M 1927M 3745M 17.2.5 cc65afd6173a b092b03f211d osd.44 osd4 running (103m) 3m ago 6M 3332M 3745M 17.2.5 cc65afd6173a 9309d93c80da osd.45 osd4 running (103m) 3m ago 6M 3172M 3745M 17.2.5 cc65afd6173a 8a47d7c155cd osd.46 osd4 running (103m) 3m ago 6M 2545M 3745M 17.2.5 cc65afd6173a 0550b66a837f osd.47 osd4 running (103m) 3m ago 6M 3010M 3745M 17.2.5 cc65afd6173a 52328564e503 osd.48 osd5 running (8w) 8m ago 8w 6054M 3745M 17.2.5 cc65afd6173a c5494c95566e osd.49 osd5 running (8w) 8m ago 8w 5648M 3745M 17.2.5 cc65afd6173a cebc3405950e osd.50 osd5 running (8w) 8m ago 8w 4898M 3745M 17.2.5 cc65afd6173a 44bd2f018476 osd.51 osd5 running (8w) 8m ago 8w 5066M 3745M 17.2.5 cc65afd6173a 008cb9d89e68 osd.52 osd5 running (8w) 8m ago 8w 6325M 3745M 17.2.5 cc65afd6173a db3d01bbfce7 osd.53 osd5 running (8w) 8m ago 8w 6040M 3745M 17.2.5 cc65afd6173a 215c70cf7ca3 osd.54 osd5 running (8w) 8m ago 8w 5147M 3745M 17.2.5 cc65afd6173a 92a4bc3eae20 osd.55 osd5 running (8w) 8m ago 8w 6590M 3745M 17.2.5 cc65afd6173a ce07fe507b47 osd.56 osd5 running (8w) 8m ago 8w 3838M 3745M 17.2.5 cc65afd6173a 540cf969419d osd.57 osd5 running (8w) 8m ago 8w 4820M 3745M 17.2.5 cc65afd6173a 89653e57e85c osd.58 osd5 running (8w) 8m ago 8w 5016M 3745M 17.2.5 cc65afd6173a c19d06b09533 osd.59 osd5 running (8w) 8m ago 8w 6547M 3745M 17.2.5 cc65afd6173a 8c387d3d06a6 osd.60 osd6 running (99m) 3m ago 6M 2050M 3745M 17.2.5 cc65afd6173a 90c1d1745480 osd.61 osd6 running (99m) 3m ago 6M 2694M 3745M 17.2.5 cc65afd6173a 32da4b4cec45 osd.62 osd6 running (99m) 3m ago 6M 2608M 3745M 17.2.5 cc65afd6173a 43788b932105 osd.63 osd6 running (99m) 3m ago 6M 4048M 3745M 17.2.5 cc65afd6173a 4a6863748bc5 osd.64 osd6 running (99m) 3m ago 6M 2542M 3745M 17.2.5 cc65afd6173a c49dd095aa64 osd.65 osd6 running (99m) 3m ago 6M 2796M 3745M 17.2.5 cc65afd6173a cdb2bed58cf3 osd.66 osd6 running (99m) 3m ago 6M 2129M 3745M 17.2.5 cc65afd6173a c17db58e047f osd.67 osd6 running (99m) 3m ago 6M 3559M 3745M 17.2.5 cc65afd6173a 46f2527c6112 osd.68 osd6 running (99m) 3m ago 6M 2552M 3745M 17.2.5 cc65afd6173a c4491c1b8de9 osd.69 osd6 running (99m) 3m ago 6M 2750M 3745M 17.2.5 cc65afd6173a 9e03c1d5351e osd.70 osd6 running (99m) 3m ago 6M 1849M 3745M 17.2.5 cc65afd6173a 3454e09aab70 osd.72 osd6 running (99m) 3m ago 8w 5368M 3745M 17.2.5 cc65afd6173a 0c1ca9a8677d # ceph orch host ls HOST ADDR LABELS STATUS iscsi1 192.168.6.166 iscsi2 192.168.6.167 mon1 192.168.7.208 mon2 192.168.7.209 mon3 192.168.7.210 osd1 192.168.7.214 osd2 192.168.7.215 osd3 192.168.7.216 osd4 192.168.7.217 osd5 192.168.7.218 osd6 192.168.7.219 11 hosts in cluster # ceph health detail HEALTH_WARN 12 stray daemon(s) not managed by cephadm; 3 stray host(s) with 35 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 12 stray daemon(s) not managed by cephadm stray daemon osd.12 on host osd2 not managed by cephadm stray daemon osd.13 on host osd2 not managed by cephadm stray daemon osd.14 on host osd2 not managed by cephadm stray daemon osd.15 on host osd2 not managed by cephadm stray daemon osd.16 on host osd2 not managed by cephadm stray daemon osd.17 on host osd2 not managed by cephadm stray daemon osd.18 on host osd2 not managed by cephadm stray daemon osd.19 on host osd2 not managed by cephadm stray daemon osd.20 on host osd2 not managed by cephadm stray daemon osd.21 on host osd2 not managed by cephadm stray daemon osd.22 on host osd2 not managed by cephadm stray daemon osd.23 on host osd2 not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 3 stray host(s) with 35 daemon(s) not managed by cephadm stray host c-osd3 has 12 stray daemons: ['osd.24', 'osd.25', 'osd.26', 'osd.27', 'osd.28', 'osd.29', 'osd.30', 'osd.31', 'osd.32', 'osd.33', 'osd.34', 'osd.35'] stray host c-osd4 has 12 stray daemons: ['osd.36', 'osd.37', 'osd.38', 'osd.39', 'osd.40', 'osd.41', 'osd.42', 'osd.43', 'osd.44', 'osd.45', 'osd.46', 'osd.47'] stray host c-osd6 has 11 stray daemons: ['osd.60', 'osd.61', 'osd.62', 'osd.63', 'osd.64', 'osd.65', 'osd.66', 'osd.67', 'osd.68', 'osd.69', 'osd.70'] Comments: 1) c-osd3, c-osd4, c-osd6 are old names (with old addresses) which are not in cluster now, as you can see from 'ceph orch ps' listing. 2) On the osd1 machine I've tried create osd daemons bych 'ceph orch', but I cannot create with one NVMe for DB and WAL together, thus I have OSD without cached DB and WAL. 3) On the osd2, there is a OSD daemons created by ceph-volume 4) On the other hosts I've renamed host, moved to another IP address, 'ceph orch rm c-osdx', 'ceph orch add osdx', in LVM changed all c-osdx tags to osdx tags, now I've changed in /var/lib/ceph/uuid/osd.x/unit.run and unit.poststop every c-osdx to osdx. And now questions: Is there a way to definitelly remove c-osd3, c-osd4 and c-osd6 from 'ceph health detail' and manage this host via 'ceph orch'? On the osd2, there is a problem to adopt OSD created by ceph-volume to ceph orchestrator: cephadm adopt --style legacy -n osd.12 Pulling container image quay.io/ceph/ceph:v17... Found online OSD at //var/lib/ceph/osd/ceph-12/fsid objectstore_type is bluestore Stopping old systemd unit ceph-osd@12... Disabling old systemd unit ceph-osd@12... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9468, in <module> main() File "/usr/sbin/cephadm", line 9456, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2135, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6582, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6755, in command_adopt_ceph os.rmdir(data_dir_src) OSError: [Errno 39] Directory not empty: '//var/lib/ceph/osd/ceph-12' Thanks a lot. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

1 0

Re: Bucket sync policy
by Yixin Jin 25 Apr '23

25 Apr '23

Actually, "bucket sync run" somehow made it worse since now the destination zone shows "bucket is caught up with source" from "bucket sync status" even though it clearly missed an object. On Monday, April 24, 2023 at 02:37:46 p.m. EDT, Yixin Jin <yjin77(a)yahoo.ca> wrote: An update: After creating and enabling the bucket sync policy, I ran "bucket sync markers" and saw that each shard had the status of "init". The run "bucket sync run" in the end marked the status to be "incremental-sync", which seems to go through full-sync stage. However, the lone object in the source zone wasn't synced over to the destination zone. I actually used gdb to walk through radosgw-admin to run "bucket sync run". It seems not to do anything for full-sync and it printed a log saying "finished iterating over all available prefixes:...", which actually broke off the do-while loop after the call to prefix_handler.revalidate_marker(&list_marker). This call returned false because it couldn't find rules from the sync pipe. I haven't drilled deeper to see why it didn't get rules, whatever it means. Nevertheless, the workaround with "bucket sync run" doesn't seem to work, at least not with Quincy. Regards,Yixin On Monday, April 24, 2023 at 12:37:24 p.m. EDT, Soumya Koduri <skoduri(a)redhat.com> wrote: On 4/24/23 21:52, Yixin Jin wrote: > Hello ceph gurus, > > We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. > > > > Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync? yes this is known_issue yet to be addressed with bucket level sync policy ( - https://tracker.ceph.com/issues/57489 ). The interim workaround to sync existing objects is to either * create new objects (or) * execute "bucket sync run" after creating/enabling the bucket policy. Please note that this issue is specific to only bucket policy but doesn't exist for sync-policy set at zonegroup level. Thanks, Soumya _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 3

Bucket sync policy
by Yixin Jin 25 Apr '23

25 Apr '23

Hello ceph gurus, We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync? I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. I also tried "bucket sync init" and "bucket sync run" via radosgw-admin. They don't seem to do what I expected. They simply mark z1 as not behind anymore but the single object still lives in z0 only. I wonder how mature this sync policy feature is for production use. Thanks,Yixin

3 2

pacific 16.2.13 point release
by Yuri Weinstein 24 Apr '23

24 Apr '23

We want to do the next urgent point release for pacific 16.2.13 ASAP. The tip of the current pacific branch will be used as a base for this release and we will build it later today. Dev leads - if you have any outstanding PRs that must be included pls merged them now. Thx YuriW

3 2

Re: Bucket sync policy
by Yixin Jin 24 Apr '23

24 Apr '23

Re-post my rely to make sure it goes through the mailing list. ----- Forwarded Message ----- From: Yixin Jin <yjin77(a)yahoo.ca>To: Soumya Koduri <skoduri(a)redhat.com>Sent: Monday, April 24, 2023 at 02:37:46 p.m. EDTSubject: Re: [ceph-users] Re: Bucket sync policy An update: After creating and enabling the bucket sync policy, I ran "bucket sync markers" and saw that each shard had the status of "init". The run "bucket sync run" in the end marked the status to be "incremental-sync", which seems to go through full-sync stage. However, the lone object in the source zone wasn't synced over to the destination zone. I actually used gdb to walk through radosgw-admin to run "bucket sync run". It seems not to do anything for full-sync and it printed a log saying "finished iterating over all available prefixes:...", which actually broke off the do-while loop after the call to prefix_handler.revalidate_marker(&list_marker). This call returned false because it couldn't find rules from the sync pipe. I haven't drilled deeper to see why it didn't get rules, whatever it means. Nevertheless, the workaround with "bucket sync run" doesn't seem to work, at least not with Quincy. Regards,Yixin On Monday, April 24, 2023 at 12:37:24 p.m. EDT, Soumya Koduri <skoduri(a)redhat.com> wrote: On 4/24/23 21:52, Yixin Jin wrote: > Hello ceph gurus, > > We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. > > > > Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync? yes this is known_issue yet to be addressed with bucket level sync policy ( - https://tracker.ceph.com/issues/57489 ). The interim workaround to sync existing objects is to either * create new objects (or) * execute "bucket sync run" after creating/enabling the bucket policy. Please note that this issue is specific to only bucket policy but doesn't exist for sync-policy set at zonegroup level. Thanks, Soumya _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

v16.2.12 Pacific (hot-fix) released
by Yuri Weinstein 24 Apr '23

24 Apr '23

We're happy to announce the 12th hot-fix release in the Pacific series. https://ceph.io/en/news/blog/2023/v16-2-12-pacific-released/ Notable Changes --------------- This is a hotfix release that resolves several performance flaws in ceph-volume, particularly during osd activation (https://tracker.ceph.com/issues/57627) Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.12.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 5a2d516ce4b134bfafc80c4274532ac0d56fc1e2

4 4

Bucket sync policy
by yjin77＠yahoo.ca 24 Apr '23

24 Apr '23

Hello, ceph gurus, We are trying multisite sync policy feature with Quincy release and we encounter something strange, which we cannot solve even after combing through the internet for clues. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-level sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. So, why wasn’t there a full sync and how can one force a full sync? I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. Thanks, Yixin

1 0

Disks are filling up
by Omar Siam 24 Apr '23

24 Apr '23

Hi list, we created a cluster for using cephfs with a kubernetes cluster. Since a few weeks now the cluster keeps filling up at an alarming rate (100 GB per day). This is while the most relevant pg is deep scrubbing and was interupted a few times. We use about 150G (du using the mounted filesystem) on the cephfs filesystem and try not to use snapshots (.snap directories "exist" but are empty). We do not understand why the pgs get bigger and bigger while cephfs stays about the same size (overwrites on files certainly happen). I suspect some snapshots mechanism. Any ideas how to debug this to stop it? Maybe we should try to speed up the deep scrubbing somehow? ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) cluster: id: ece0290c-cd32-11ec-a0e2-005056a9dd02 health: HEALTH_WARN 1 MDSs report slow metadata IOs 3 nearfull osd(s) 13 pool(s) nearfull services: mon: 3 daemons, quorum acdh-gluster-hdd3,acdh-gluster-hdd1,acdh-gluster-hdd2 (age 3d) mgr: acdh-gluster-hdd3.kzsplh(active, since 5d), standbys: acdh-gluster-hdd2.kiotbg, acdh-gluster-hdd1.ywgyfx mds: 1/1 daemons up, 2 standby osd: 3 osds: 3 up (since 4d), 3 in (since 7w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 292 pgs objects: 167.25M objects, 1.2 TiB usage: 3.3 TiB used, 1.2 TiB / 4.5 TiB avail pgs: 290 active+clean 1 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 58 MiB/s rd, 3.6 MiB/s wr, 51 op/s rd, 148 op/s wr rancher-ceph-fs - 227 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mds.acdh-gluster-hdd1.pqydya Reqs: 68 /s 793k 792k 102k 210k POOL TYPE USED AVAIL rancherFsPoolMetadata metadata 160G 329G rancherFsPoolDefaultData data 2268k 329G rancherFsPoolMainData data 2584G 658G STANDBY MDS ceph-mds.acdh-gluster-hdd2.zfleqe ceph-mds.acdh-gluster-hdd3.etaobl MDS version: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) (rancherFsPoolMainData is a 2+1 erasure encoded pool) --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 TOTAL 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 331 GiB rancher-rbd-erasure 2 32 8.4 GiB 2.16k 13 GiB 1.25 661 GiB rancher-rbd-meta 3 32 55 B 11 36 KiB 0 331 GiB rancherFsPoolMetadata 4 32 53 GiB 5.18M 160 GiB 13.88 331 GiB rancherFsPoolDefaultData 5 1 29 KiB 80.00M 2.2 MiB 0 331 GiB rancherFsPoolMainData 6 1 1.7 TiB 82.08M 2.5 TiB 72.23 661 GiB .rgw.root 7 32 1.3 KiB 4 48 KiB 0 331 GiB default.rgw.log 8 32 3.6 KiB 209 408 KiB 0 331 GiB default.rgw.control 9 32 0 B 8 0 B 0 331 GiB default.rgw.meta 10 32 3.8 KiB 11 124 KiB 0 331 GiB default.rgw.buckets.index 11 32 2.4 MiB 33 7.2 MiB 0 331 GiB default.rgw.buckets.non-ec 12 32 0 B 0 0 B 0 331 GiB default.rgw.buckets.data 14 1 55 GiB 16.57k 83 GiB 7.70 661 GiB HEALTH_WARN 1 MDSs report slow metadata IOs; 3 nearfull osd(s); 13 pool(s) nearfull [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.ceph-mds.acdh-gluster-hdd1.pqydya(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 306 secs [WRN] OSD_NEARFULL: 3 nearfull osd(s) osd.0 is near full osd.2 is near full osd.3 is near full [WRN] POOL_NEARFULL: 13 pool(s) nearfull pool 'device_health_metrics' is nearfull pool 'rancher-rbd-erasure' is nearfull pool 'rancher-rbd-meta' is nearfull pool 'rancherFsPoolMetadata' is nearfull pool 'rancherFsPoolDefaultData' is nearfull pool 'rancherFsPoolMainData' is nearfull pool '.rgw.root' is nearfull pool 'default.rgw.log' is nearfull pool 'default.rgw.control' is nearfull pool 'default.rgw.meta' is nearfull pool 'default.rgw.buckets.index' is nearfull pool 'default.rgw.buckets.non-ec' is nearfull pool 'default.rgw.buckets.data' is nearfull (near full is set to 0.66) -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

1 0

Troubleshooting cephadm OSDs aborting start
by André Gemünd 24 Apr '23

24 Apr '23

Dear Ceph-users, we have trouble with a Ceph cluster after a full shutdown. A couple of OSDs don't start anymore, exiting with SIGABRT very quickly. With debug logs and lots of work (I find cephadm clusters hard to debug btw) we received the following stack trace: debug -16> 2023-04-14T11:52:17.617+0000 7f10ab4d2700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/osd/PGLog.h: In function 'void PGLog::IndexedLog::add(const pg_log_entry_t&, bool)' thread 7f10ab4d2700 time 2023-04-14T11:52:17.614095+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/osd/PGLog.h: 607: FAILED ceph_assert(head.version == 0 || e.version.version > head.version) ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x55b2dafc7b7e] 2: /usr/bin/ceph-osd(+0x56ad98) [0x55b2dafc7d98] 3: (bool PGLog::append_log_entries_update_missing<pg_missing_set<true> >(hobject_t const&, std::__cxx11::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)22, pg_log_entry_t> > const&, bool, PGLog::IndexedLog*, pg_missing_set<true>&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0xc19) [0x55b2db1bb6b9] 4: (PGLog::merge_log(pg_info_t&, pg_log_t&&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xee2) [0x55b2db1adf22] 5: (PeeringState::merge_log(ceph::os::Transaction&, pg_info_t&, pg_log_t&&, pg_shard_t)+0x75) [0x55b2db33c165] 6: (PeeringState::Stray::react(MLogRec const&)+0xcc) [0x55b2db37adec] 7: (boost::statechart::simple_state<PeeringState::Stray, PeeringState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd5) [0x55b2db3a6e65] 8: (boost::statechart::state_machine<PeeringState::PeeringMachine, PeeringState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x5b) [0x55b2db18ef6b] 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PeeringCtx&)+0x2d1) [0x55b2db1839e1] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x29c) [0x55b2db0fde5c] 11: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x55b2db32d0e6] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x55b2db0efd48] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55b2db7615b4] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b2db764254] 15: /lib64/libpthread.so.0(+0x817f) [0x7f10cef1117f] 16: clone() debug -15> 2023-04-14T11:52:17.618+0000 7f10b64e8700 3 osd.70 72507 handle_osd_map epochs [72507,72507], i have 72507, src has [68212,72507] debug -14> 2023-04-14T11:52:17.619+0000 7f10b64e8700 3 osd.70 72507 handle_osd_map epochs [72507,72507], i have 72507, src has [68212,72507] debug -13> 2023-04-14T11:52:17.619+0000 7f10ac4d4700 5 osd.70 pg_epoch: 72507 pg[18.7( v 64162'106 (0'0,64162'106] local-lis/les=72506/72507 n=14 ec=17104/17104 lis/c=72506/72480 les/c/f=72507/72481/0 sis=72506 pruub=9.160680771s) [70,86,41] r=0 lpr=72506 pi=[72480,72506)/1 crt=64162'106 lcod 0'0 mlcod 0'0 active+wait pruub 12.822580338s@ mbc={}] exit Started/Primary/Active/Activating 0.011269 7 0.000114 # ceph versions { "mon": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "osd": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 92 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 103 } } Another thing is that things like `ceph -s`, `ceph osd tree`, `rbd ls`, etc. work, but `ceph orch ps` (or generally any orch commands) simply hang forever, seemingly in a futex waiting on a socket to the mons. If anyone has any ideas how we could get those OSDs back online, I'd be very grateful for any hints. I'm also on slack. Greetings -- André Gemünd, Leiter IT / Head of IT Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemuend(a)scai.fraunhofer.de Tel: +49 2241 14-4199 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

2 2

How to replace an HDD in a OSD with shared SSD for DB/WAL
by Tao LIU 23 Apr '23

23 Apr '23

HI, I build a Ceph Cluster with cephadm. Every cehp node has 4 OSDs. These 4 OSD were build with 4 HDD (block) and 1 SDD (DB). At present , one HDD is broken, and I am trying to replace the HDD,and build the OSD with the new HDD and the free space of the SDD. I did the follows: #ceph osd stop osd.23 #ceph osd out osd.23 #ceph osd crush remove osd.23 #ceph osd rm osd.23 #ceph orch daemon rm osd.23 --force #lvremove /dev/ceph-ae21e618-601e-4273-9185-99180edb8453/osd-block-96eda371-1a3f-4139-9123-24ec1ba362c4 #wipefs -af /dev/sda #lvremove /dev/ceph-e50203a6-8b8e-480f-965c-790e21515395/osd-db-70f7a032-cf2c-4964-b979-2b90f43f2216 #ceph orch daemon add osd compute11:data_devices=/dev/sda,db_devices=/dev/sdc,osds_per_device=1 The OSD can be built, but is always down. Is there anyting that I missed during the building? Thank you very much! Regards, LIUTao

3 2

radosgw-admin bucket rm fails
by James Turner 22 Apr '23

22 Apr '23

I've got buckets that are visible to radosgw-admin bucket list but not to an S3 client and the delete fails.. e.g.: radosgw-admin bucket rm --bucket=appsshared1 2023-04-22T19:43:55.705-0400 7f9a87616680 0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-2 2023-04-22T19:43:55.705-0400 7f9a87616680 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -2 for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) Similarly, trying to delete the owning user fails: radosgw-admin user rm --uid=apps_shared_user --purge-data could not remove user: unable to remove user, unable to delete user data 2023-04-22T19:54:34.266-0400 7f84c598c680 0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-2 2023-04-22T19:54:34.266-0400 7f84c598c680 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -2 for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) As does trying to fix the bucket: radosgw-admin bucket check --bucket appsshared1 --fix 2023-04-22T19:56:17.510-0400 7f981eddd680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) failed The release is quincy - Any ideas on how to approach the problem? -- Jim Turner

1 0

advise on adding RGW and NFS/iSCSI on proxmox
by MartijnF 22 Apr '23

22 Apr '23

Hello, i'm maintaining a 3 small infrastructures in NL. 7 primary, 4 secondary and 3 testing proxmox nodes (hyperconverged). Till now we only use the embedded RBD facilities of proxmox using ceph 17.2.5 To facilitate S3/swift and persistent storage to our internal k8s cluster i am looking around for best-practices like: - should i install it on-top of proxmox-hosts or create a few ha-VM's with public and private lan-segment to avoid upgrade errors. - can i use the new ceph-dashboard to create and install mentioned services needed? Should i virtualize a proxmox or other OS for this? I can imagine it is not their primary focus. A lot of information can be found when using google-foo but it is hard to find answers reflecting my setup. Thanks in advance for any pointers! Martijn

1 0

[sync policy] multisite bucket full sync
by yjin77＠yahoo.ca 21 Apr '23

21 Apr '23

Hi folks, We are trying multisite sync policy feature with Quincy release and we encounter something strange. Maybe our understanding of sync policy is incorrect. I hope the community could help us uncover the mystery. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm "world", a single zonegroup "zg" and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with status “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy status is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-level sync policy with status “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind for data sync, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. My questions are: 1. With bucket-specific sync policy, why wasn’t there a full sync and how can one force a full sync? 2. Does the sync policy only work datalogs, without which no action it may take? I wonder if it would explain why it didn't sync even though it knows it is behind? BTW, I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. Thanks in advance, Yixin

1 0

Re: pg_autoscaler using uncompressed bytes as pool current total_bytes triggering false POOL_TARGET_SIZE_BYTES_OVERCOMMITTED warnings?
by Christian Rohmann 21 Apr '23

21 Apr '23

Hey ceph-users, may I ask (nag) again about this issue? I am wondering if anybody can confirm my observations? I raised a bug https://tracker.ceph.com/issues/54136, but apart from the assignment to a dev a while ago here was not response yet. Maybe I am just holding it wrong, please someone enlighten me. Thank you and with kind regards Christian On 02/02/2022 20:10, Christian Rohmann wrote: > > Hey ceph-users, > > > I am debugging a mgr pg_autoscaler WARN which states a > target_size_bytes on a pool would overcommit the available storage. > There is only one pool with value for target_size_bytes (=5T) defined > and that apparently would consume more than the available storage: > > --- cut --- > # ceph health detail > HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes > [WRN] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED: 1 subtrees have > overcommitted pool target_size_bytes > Pools ['backups', 'images', 'device_health_metrics', '.rgw.root', > 'redacted.rgw.control', 'redacted.rgw.meta', 'redacted.rgw.log', > 'redacted.rgw.otp', 'redacted.rgw.buckets.index', > 'redacted.rgw.buckets.data', 'redacted.rgw.buckets.non-ec'] overcommit > available storage by 1.011x due to target_size_bytes 15.0T on pools > ['redacted.rgw.buckets.data']. > --- cut --- > > > But then looking at the actual usage it seems strange that 15T (5T * 3 > replicas) should not fit onto the remaining 122 TiB AVAIL: > > > --- cut --- > # ceph df detail > --- RAW STORAGE --- > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 293 TiB 122 TiB 171 TiB 171 TiB 58.44 > TOTAL 293 TiB 122 TiB 171 TiB 171 TiB 58.44 > > --- POOLS --- > POOL ID PGS STORED (DATA) (OMAP) > OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS > QUOTA BYTES DIRTY USED COMPR UNDER COMPR > backups 1 1024 92 TiB 92 TiB 3.8 MiB > 28.11M 156 TiB 156 TiB 11 MiB 64.77 28 TiB N/A > N/A N/A 39 TiB 123 TiB > images 2 64 1.7 TiB 1.7 TiB 249 KiB > 471.72k 5.2 TiB 5.2 TiB 748 KiB 5.81 28 TiB N/A > N/A N/A 0 B 0 B > device_health_metrics 19 1 82 MiB 0 B 82 > MiB 43 245 MiB 0 B 245 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > .rgw.root 21 32 23 KiB 23 KiB 0 B > 25 4.1 MiB 4.1 MiB 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.control 22 32 0 B 0 B 0 B > 8 0 B 0 B 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.meta 23 32 1.7 MiB 394 KiB 1.3 > MiB 1.38k 237 MiB 233 MiB 3.9 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.log 24 32 53 MiB 500 KiB 53 > MiB 7.60k 204 MiB 47 MiB 158 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.otp 25 32 5.2 KiB 0 B 5.2 > KiB 0 16 KiB 0 B 16 KiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.buckets.index 26 32 1.2 GiB 0 B 1.2 > GiB 7.46k 3.5 GiB 0 B 3.5 GiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.buckets.data 27 128 3.1 TiB 3.1 TiB 0 B > 3.53M 9.5 TiB 9.5 TiB 0 B 10.11 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.buckets.non-ec 28 32 0 B 0 B 0 B > 0 0 B 0 B 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > --- cut --- > > > I then looked at how those values are determined at > https://github.com/ceph/ceph/blob/9f723519257eca039126a20aa6a2a7d2dbfb5dba/…. > Apparently "total_bytes" are compared with the capacity of the > root_map. I added a debug line and found that the total in my cluster > was already at: > > total=325511007759696 > > so in excess of 300 TiB - Looking at "ceph df" again this usage seems > strange. > > > > Looking at how this total is calculated at > https://github.com/ceph/ceph/blob/9f723519257eca039126a20aa6a2a7d2dbfb5dba/…, > you see that the larger value (max) of "actual_raw_used" vs. > "target_bytes*raw_used_rate" is determined and then summed up. > > > I dumped the values for all pools my cluster with yet another line of > debug code: > > ---cut --- > pool_id 1 - actual_raw_used=303160109187420.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 2 - actual_raw_used=5714098884702.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 19 - actual_raw_used=256550760.0, target_bytes=0 raw_used_rate=3.0 > pool_id 21 - actual_raw_used=71433.0, target_bytes=0 raw_used_rate=3.0 > pool_id 22 - actual_raw_used=0.0, target_bytes=0 raw_used_rate=3.0 > pool_id 23 - actual_raw_used=5262798.0, target_bytes=0 raw_used_rate=3.0 > pool_id 24 - actual_raw_used=162299940.0, target_bytes=0 raw_used_rate=3.0 > pool_id 25 - actual_raw_used=16083.0, target_bytes=0 raw_used_rate=3.0 > pool_id 26 - actual_raw_used=3728679936.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 27 - actual_raw_used=10035209699328.0, > target_bytes=5497558138880 raw_used_rate=3.0 > pool_id 28 - actual_raw_used=0.0, target_bytes=0 raw_used_rate=3.0 > --- cut --- > > > All values but those of pool_id 1 (backups) make sense. For backups > it's just reporting a MUCH larger actual_raw_used value than what is > shown via ceph df. > The only difference of that pool compared to the others is the enabled > compression: > > > --- cut --- > # ceph osd pool get backups compression_mode > compression_mode: aggressive > --- cut --- > > > Apparently there already was a similar issue > (https://tracker.ceph.com/issues/41567) with a resulting commit > (https://github.com/ceph/ceph/commit/dd6e752826bc762095be4d276e3c1b8d31293eb0) > > changing which from "bytes_used" to the "stored" field for > "pool_logical_used". > > But how does that take compressed (away) data into account? Does > "bytes_used" count all the "stored" bytes, summing up all uncompressed > bytes for pools with compression? > This surely must be a bug then, as those bytes are not really > "actual_raw_used". > > > > I was about to raise a bug, but I wanted to ask here on the ML first > if I misunderstood the mechanisms at play here. > Thanks and with kind regards, > > > Christian >

1 0

17.2.6 dashboard: unable to get RGW dashboard working
by Michel Jouvin 21 Apr '23

21 Apr '23

Hi, I just upgraded in 17.2.6 but in fact I had the same problem in 16.2.10. I'm trying to configure the Ceph dashboard to monitor the RGWs (object gateways used as S3 gw). Our cluster has 2 RGW realms (eros, fink) with 1 zonegroup per realm (p2io-eros and p2io-fink respectively) and 1 zone per zonegroup (eros-z1 and fink-z1 respectively). RGW have been configured without SSL due to the bug encountered in 16.2.10 (not sure if it still applies, use of IP addresses that don't match the certificate). There is a dashboard user in each realm and its access/secret keys are matching what is in mgr/dashboard/RGW_API_ACCESS_KEY and mgr/dashboard/RGW_API_SECRET_KEY (a dict with one entry per realm). Every attempt to access the RGW daemons section of the dashboard results in : ----- The Object Gateway Service is not configured Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx00000b4c2212597168bf0-00644116e3' b'-edf7c3-eros-z1","HostId":"edf7c3-eros-z1-ijclab"}') ---- In the mgr logs, I have not been able to find more than this error. In the RGW logs, for every dashboard attempt, I find: ------ Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: ====== starting new request req=0x7f2dcdb0b710 ===== Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: req 13025009517641239536 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: ====== req done req=0x7f2dcdb0b710 op status=0 http_status=403 latency=0.000000000s ====== Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: beast: 0x7f2dcdb0b710: 134.158.194.10 - - [20/Apr/2023:10:41:39.843 +0000] "GET /admin/metadata/user?myself HTTP/1.1" 403 126 - "python-requests/2.20.0" - latency=0.000000000s ------ In the logs above, I find a little bit strange the "HostID" value ending with "eros-z1-ijclab". I'd expected it to be the zone name "eros-z1". The zone has been renamed a couple of times and it may be an old name, despite the zone information in radosgw-admin shows nothing with this name (in particular the API name has been renamed to match the zone name). Anway the RGW works perfectly and the dashboard request seems to reach the RGW so I don't have the feeling it is important... Any suggestion on how to troubleshoot what happens would be appreciated. Thanks in advance. Best regards, Michel

1 1

deploying Ceph using FQDN for MON / MDS Services
by Lokendra Rathour 21 Apr '23

21 Apr '23

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

7 20

How to replace an HDD in a OSD with shared SSD for DB/WAL
by enochlew＠gmail.com 21 Apr '23

21 Apr '23

HI, I build a Ceph Cluster with cephadm. Every cehp node has 4 OSDs. These 4 OSD were build with 4 HDD (block) and 1 SDD (DB). At present , one HDD is broken, and I am trying to replace the HDD,and build the OSD with the new HDD and the free space of the SDD. I did the follows: #ceph osd stop osd.23 #ceph osd out osd.23 #ceph osd crush remove osd.23 #ceph osd rm osd.23 #ceph orch daemon rm osd.23 --force #lvremove /dev/ceph-ae21e618-601e-4273-9185-99180edb8453/osd-block-96eda371-1a3f-4139-9123-24ec1ba362c4 #wipefs -af /dev/sda #lvremove /dev/ceph-e50203a6-8b8e-480f-965c-790e21515395/osd-db-70f7a032-cf2c-4964-b979-2b90f43f2216 #ceph orch daemon add osd compute11:data_devices=/dev/sda,db_devices=/dev/sdc,osds_per_device=1 The OSD can be built, but is always down. Is there anyting that I missed during the building? Thank you very much! Regards, LIUTao

1 0

I am unable to execute 'rbd map xxx' as it returns the error 'rbd: map failed: (5) Input/output error'.
by siriusa51＠outlook.com 21 Apr '23

21 Apr '23

$ rbd map xxx rbd: sysfs write failed 2023-04-21 11:29:13.786418 7fca1bfff700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2023-04-21 11:29:13.786456 7fca1b7fe700 -1 librbd::ImageState: 0x55a60108a040 failed to open image: (5) Input/output error rbd: error opening image xxx: (5) Input/output error In some cases useful info is found in syslog - try "dmesg | tail" or so. rbd: map failed: (5) Input/output error the command 'dmesg | tail' did not show any useful results. and other command: $ rbd info xxx 2023-04-21 11:33:10.223701 7f3547fff700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2023-04-21 11:33:10.223739 7f35477fe700 -1 librbd::ImageState: 0x5647d5cfeeb0 failed to open image: (5) Input/output error rbd: error opening image xxx: (5) Input/output error i know header id is c2c061579478fe ,so i get omapvals: $ rados -p rbd listomapvals rbd_header.c2c061579478fe features value (8 bytes) : 00000000 01 00 00 00 00 00 00 00 |........| 00000008 object_prefix value (27 bytes) : 00000000 17 00 00 00 72 62 64 5f 64 61 74 61 2e 63 32 63 |....rbd_data.c2c| 00000010 30 36 31 35 37 39 34 37 38 66 65 |061579478fe| 0000001b order value (1 bytes) : 00000000 16 |.| 00000001 size value (8 bytes) : 00000000 00 00 00 00 71 02 00 00 |....q...| 00000008 snap_seq value (8 bytes) : 00000000 00 00 00 00 00 00 00 00 |........| 00000008 I scanned all existing data blocks using the command 'rados -p rbd list | grep c2c061579478fe', and found many 'rbd_data' blocks. I suspect that the loss of important blocks caused this situation. Currently, my plan is to obtain all 'rbd' blocks, concatenate them into an XFS file, and then perform a repair. However, as the data size is huge, I am still experimenting with this method. Are there any other feasible methods to repair this data? rbd_data: rbd_data.c2c061579478fe.0000000000000000 rbd_data.c2c061579478fe.0000000000000001 rbd_data.c2c061579478fe.0000000000000002 rbd_data.c2c061579478fe.0000000000000003 rbd_data.c2c061579478fe.0000000000000004 rbd_data.c2c061579478fe.0000000000000005 rbd_data.c2c061579478fe.0000000000000006 rbd_data.c2c061579478fe.0000000000000007 rbd_data.c2c061579478fe.0000000000000008 rbd_data.c2c061579478fe.0000000000000009 rbd_data.c2c061579478fe.000000000000000a rbd_data.c2c061579478fe.000000000000000b rbd_data.c2c061579478fe.000000000000000c rbd_data.c2c061579478fe.000000000000000d rbd_data.c2c061579478fe.000000000000000e rbd_data.c2c061579478fe.000000000000000f rbd_data.c2c061579478fe.0000000000000010 rbd_data.c2c061579478fe.0000000000000011 ... rbd_data.c2c061579478fe.000000000009c3f6 rbd_data.c2c061579478fe.000000000009c3f7 rbd_data.c2c061579478fe.000000000009c3fa rbd_data.c2c061579478fe.000000000009c3fb rbd_data.c2c061579478fe.000000000009c3fc rbd_data.c2c061579478fe.000000000009c3fd rbd_data.c2c061579478fe.000000000009c3fe rbd_data.c2c061579478fe.000000000009c3ff rbd_header.c2c061579478fe thanks

1 0

Can I delete rgw log entries?
by Richard Bade 20 Apr '23

20 Apr '23

Hi Everyone, I've been having trouble finding an answer to this question. Basically I'm wanting to know if stuff in the .log pool is actively used for anything or if it's just logs that can be deleted. In particular I was wondering about sync logs. In my particular situation I have had some tests of zone sync setup, but now I've removed the secondary zone and pools. My primary zone is filled with thousands of logs like this: data_log.71 data.full-sync.index.e2cf2c3e-7870-4fc4-8ab9-d78a17263b4f.47 meta.full-sync.index.7 datalog.sync-status.shard.e2cf2c3e-7870-4fc4-8ab9-d78a17263b4f.13 bucket.sync-status.f3113d30-ecd3-4873-8537-aa006e54b884:{bucketname}:default.623958784.455 I assume that because I'm not doing any sync anymore I can delete all the sync related logs? Is anyone able to confirm this? What about if the sync is running? Are these being written and read from and therefore must be left alone? It seems like these are more of a status than just a log and that deleting them might confuse the sync process. If so, does that mean that the log pool is not just output that can be removed as needed? Are there perhaps other things in there that need to stay? Regards, Richard

3 3

Be careful with primary-temp to balance primaries ...
by Stefan Kooman 20 Apr '23

20 Apr '23

Hi, A word of caution for Ceph operators out there. Be careful with "ceph osd primary-temp" command. TL;DR: with primary_temp active, a CRUSH change might CRASH your OSDs ... and they won't come back online after a restart (in almost all cases). The bug is described in this tracker [1], and fixed with this PR [2] (thanks Igor!). The longer story is that we were inspired by the work done on the read-balancer [3] and wondered if we could leverage this on older clusters. It turned out this is indeed possible by using the "primary-temp" command, instead of "pg-primary-temp" that will be available from Reef onward. We compiled a main version of the "osdmaptool", fed it OSD maps from pacific clusters and have it calculate the optimal primary PG distributions. Then we replaced the pg-primary-temp command with "primary-temp" and applied the commands. That worked as expected. However, we hit a bug [1] in the Ceph code that could not handle a situation when there were changes in the CRUSH map when primary_temp where active. We added a new storage node to the cluster that triggered this condition as soon as we put it in the proper failure domain. It tried to make an OSD primary that was not in the active set anymore, and hence crashed (with a Segmentation Fault most often, or aborted). This resulted in multiple (many) OSD crashes across the failure domains and basically took down the whole cluster. If the Reef / main read-balancer code can suffer from the same bug is as of yet unknown (at least to us). We will try to build a Reef test cluster and find out. For those of you who want to know how we handled this incident can read the RFO ([4] in dutch, [5] in English). Gr. Stefan [1]: https://tracker.ceph.com/issues/59491?next_issue_id=59490 [2]: https://github.com/ceph/ceph/pull/51160 [3]: https://github.com/ljflores/ceph_read_balancer_2023 [4]: https://www.bit.nl/uploads/images/PDF-Files/RFO-20230314-185335.pdf [5]: https://www.bit.nl/uploads/images/PDF-Files/2023.04.20%20RFO_Ceph Cluster_185335_EN.pdf

2 1

User + Dev Monthly Meetup cancelled
by Laura Flores 20 Apr '23

20 Apr '23

Hi users, The User + Dev monthly meetup is cancelled today since some folks are still at or returning from Cephalocon. See you next month! - Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

quincy user metadata constantly changing versions on multisite slave with radosgw roles
by Christopher Durham 20 Apr '23

20 Apr '23

Hi, I am using 17.2.6 on rocky linux for both the master and the slave site I noticed that: radosgw-admin sync status often shows that the metadata sync is behind a minute or two on the slave. This didn't make sense, as the metadata isn't changing as far as I know. radosgw-admin mdlog list (on slave) showed me that there were user changes in metadata very often. After doing a little research, here is a scenario I was able to develop: 1. user continually writes to a bucket he owns, pointing his aws cli (for this test) to the master side endpoint as specified in ~/.aws/config while this is running, do the following on the slave: radosgw-admin metadata get user:<username> This always shows the same result for the user, no changes. The data gets to the slave side bucket. 2. Restart the continual copy, but this time use a role that the user is a member of via profile in ~/.aws/credentials and .~/aws/config, again writing to the master endpoint as specified in ~/.aws/config aws --profile <profile> s3 cp <file> s3://<bucket>/file where profile is set up to use a role definiiton. The data gets to the bucket on both sides. I do not have access if I do not use the role profille (to confirm I set it up right) However, while doing this second test, if I continually do: radosgw-admin metadata get user:<username> on the slave, I see a definite increase in versions. Here is a section of the json output: "key": "user:<username>", "ver": { "tag": "somestring", "ver:" 12145} the 12145 value increases over and over again, and the mtime value in the json output increases too based on the current date. (not shown here). The same value, when queried on the master side, remains 1, and the mtime value is the date the user was created or last changed by an admin. If I write a file only once, the vers value increases by 1 too, but not sure if the increase in vers is ncessarily 1:1 with the number of writes. This seems to be the source of my continual metadata lag. Am I missing something? I suspect that this has been happening for awhile and not specific to 17.2.6 as I just upgraded and the ver value is over 12000 for the user that I discovered. (I used a python script to sync roles between master and slave prior to 17.2.6. Now roles are replicated in 17.2.6). -Chris

2 1

Some hint for a DELL PowerEdge T440/PERC H750 Controller...
by Marco Gaiarin 20 Apr '23

20 Apr '23

We are testing an experimental Ceph cluster with server and controller at subject. The controller have not an HBA mode, but only a 'NonRAID' mode, come sort of 'auto RAID0' configuration. We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well, and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad (particulary, very low IOPS). There's some hint for disk/controller configuration/optimization? Thanks. -- Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso... mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)

7 18

[ceph 17.2.6] unable to create rbd snapshots for images with erasure code data-pool
by Reto Gysi 20 Apr '23

20 Apr '23

Dear Ceph Users, After upgrading from version 17.2.5 to 17.2.6 I no longer seem to be able to create snapshots of images that have an erasure coded datapool. root@zephir:~# rbd snap create ceph-dev@backup_20230417 Creating snap: 10% complete...failed. rbd: failed to create snapshot: (95) Operation not supported root@zephir:~# rbd info ceph-dev rbd image 'ceph-dev': size 10 GiB in 2560 objects order 22 (4 MiB objects) snapshot_count: 11 id: d2f3d287f13c7b data_pool: ecpool_hdd block_name_prefix: rbd_data.7.d2f3d287f13c7b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Wed Nov 23 17:01:03 2022 access_timestamp: Sun Apr 16 17:20:58 2023 modify_timestamp: Wed Nov 23 17:01:03 2022 root@zephir:~# Before the upgrade I was able to create snapshots of this pool: SNAPID NAME SIZE PROTECTED TIMESTAMP 1538 ceph-dev_2023-03-05T02:00:09.030+01:00 10 GiB Sun Mar 5 02:00:14 2023 1545 ceph-dev_2023-03-06T02:00:03.832+01:00 10 GiB Mon Mar 6 02:00:05 2023 1903 ceph-dev_2023-04-05T03:22:01.315+02:00 10 GiB Wed Apr 5 03:22:02 2023 1909 ceph-dev_2023-04-05T03:35:56.748+02:00 10 GiB Wed Apr 5 03:35:57 2023 1915 ceph-dev_2023-04-05T03:37:23.778+02:00 10 GiB Wed Apr 5 03:37:24 2023 1930 ceph-dev_2023-04-06T02:00:06.159+02:00 10 GiB Thu Apr 6 02:00:07 2023 1940 ceph-dev_2023-04-07T02:00:05.913+02:00 10 GiB Fri Apr 7 02:00:06 2023 1952 ceph-dev_2023-04-08T02:00:06.534+02:00 10 GiB Sat Apr 8 02:00:07 2023 1964 ceph-dev_2023-04-09T02:00:06.430+02:00 10 GiB Sun Apr 9 02:00:07 2023 2003 ceph-dev_2023-04-11T02:00:09.750+02:00 10 GiB Tue Apr 11 02:00:10 2023 2014 ceph-dev_2023-04-12T02:00:09.528+02:00 10 GiB Wed Apr 12 02:00:10 2023 root@zephir:~# I have looked through the release notes of 17.2.6 but couldn't find anything obvious regarding rbd and ec pools. Does anyone else have this problem? Do I need to change some config setting, or was this feature disabled or is it a bug? Ceph version info: root@zephir:~# ceph orch upgrade check --ceph_version 17.2.6 { "needs_update": {}, "non_ceph_image_daemons": [ "promtail.debian", "node-exporter.debian", "promtail.zephir", "grafana.zephir", "node-exporter.zephir", "prometheus.zephir", "loki.zephir", "alertmanager.zephir" ], "target_digest": " quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "target_id": "9cea3956c04b2d889b91b58f957577fcb4eacd3852df073e3e2567f159fcdbf8", "target_name": "quay.io/ceph/ceph:v17.2.6", "target_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", "up_to_date": [ "iscsi.rbd.debian.ijztzu", "mds.jellyfin.debian.dcsocv", "mon.debian", "osd.13", "osd.6", "mds.backups.debian.runngh", "mds.rgysi.debian.uhgqen", "crash.debian", "mgr.debian.sookxi", "iscsi.rbd.zephir.viqahd", "osd.1", "mds.jellyfin.zephir.iqywsn", "osd.12", "osd.7", "osd.2", "crash.zephir", "rgw.default.zephir.jqmick", "mds.backups.zephir.ygigch", "osd.0", "osd.4", "mon.zephir", "mgr.zephir.enywvy", "mds.rgysi.zephir.diylss", "osd.3", "osd.10", "osd.5", "osd.8", "osd.11" ] } root@zephir:~# root@zephir:~# rbd --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) root@zephir:~# Cheers Reto Gysi

5 21

Ceph rgw ldap user acl and quotas
by Ignazio Cassano 19 Apr '23

19 Apr '23

Hello Everyone, We are going to install ceph object storage with ldap autenticazione. We would like to know if acl and quotas on objects and buckets work fine with ldap users. Thanks Ignazio

1 0

Unprivileged Ceph containers
by Stephen Smith6 19 Apr '23

19 Apr '23

Hey folks – just thought I’d ask here – today cephadm creates unit files for Ceph containers with --privileged in the “podman run” call. I saw https://github.com/ceph/ceph-container/blob/main/src/daemon/README.md but I’m not sure how this relates to Ceph when deployed with cephadm? Thanks, Eric

1 0

upgrading from el7 / nautilus
by Marc 19 Apr '23

19 Apr '23

Sorry for addressing this again. But I think there are quite a few still with Nautilus, that are planning such upgrade. Nautilus is currently available for el7, el8 Octopus is currently available for el7, el8 Pacific is currently available for el8, el9 Quincy is currently available for el8, el9 el7 eol 2024 I would like to end up at el9 with a the Pacific release that does not have these performance issues[1] However with the currently offered ceph releases I am more or less forced to an upgrade path of: upgrade hosts from el7 -> el8, upgrade from nautilus -> pacific, upgrade el8 -> el9 - disadvantage upgrading hosts with some co-located services to el8 and then again to el9 - disadvantage having to wait with el9 upgrade until pacific performance issue is fixed. the alternative I have been looking at is building Nautilus for el9 (centos9 stream) which seems to go ok (site-test-packages is not building though). This would result in a more efficient upgrade of: upgrade hosts from el7 -> el9, upgrade from nautilus -> pacific - advantage is only 1 os upgrade. - advantage is I can stay on Nautilus until the pacifc performance issue is fixed. Up until now I can't remember having to much issues with building things from source. However with this storage I would like to have more reassurance. 1. can anyone share experiences with building for different os? 2. what is with all these build warnings, is this common? 3. is it possible that @ceph they can build these rpms and have this go through their default (automated) testing/deploy procedures? I assume with such nice testing facilities this should not take that much time, not? 4. Can anyone recommend some testing procedures? I have a 3 node test cluster. I was thinking of upgrading one host to el9 and run these el9 Nautilus rpms on it, see how it holds. ps. I am not really interested yet in adding complexity with podman. pps. Would be nice if someone from ceph could advice on this. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg19012.html

2 1

HBA or RAID-0 + BBU
by Murilo Morais 19 Apr '23

19 Apr '23

Good evening everyone! Guys, about the P420 RAID controller, I have a question about the operation mode: What would be better: HBA or RAID-0 with BBU (active write cache)? Thanks in advance!

5 4

pacific el7 rpms
by Marc 19 Apr '23

19 Apr '23

Is there ever going to be rpms in https://download.ceph.com/rpm-pacific/el7/

2 2

Consequence of maintaining hundreds of clones of a single RBD image snapshot
by Eyal Barlev 19 Apr '23

19 Apr '23

Hello, My use-case involves creating hundreds of clones (~1,000) of a single RBD image snapshot. I assume watchers exist for each clone, due to the copy-on-write nature of clones. Should I expect a penalty for maintaining such a large number of clones: cpu, memory, performance? If such penalty does exist, we might opt to flatten some of the clones. Is consistency guaranteed during the flattening process? In other words, can I write to a clone while it is being flattened? Perspectivus

3 2

metadata sync
by Christopher Durham 17 Apr '23

17 Apr '23

Hi, After doing a 'radosgw-admin metadata sync init' on a secondary site in a 2-cluster multisite configuration, (See my previous post on doing a quincy upgade with sts roles manually synced between primary and secondary), and letting it sync all metadata from a master zone, I get the following as a large omap object warning on the secondary site: (from ceph health detail): "1 large objects found in pool <zone>.rgw.log" And in ceph.log: "Large omap object found. Object: 87.adcf1188:::meta.full-sync.index.0:head PG 87.abcdfdff (87.0) Key count: 1234567 Size (bytes) 0" I know about large omap objects, and because this is in the log pool, I should be able to delete it with the rados utility. (And do a deep scrub on the pg. Am I correct? Looks like old bucket metadata is still there even though the buckets have been deleted. -Chris

1 0