April 2023 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Re: increasing number of (deep) scrubs
by Frank Schilder 13 Dec '23

13 Dec '23

Hi Dan, thanks for your answer. I don't have a problem with increasing osd_max_scrubs (=1 at the moment) as such. I would simply prefer a somewhat finer grained way of controlling scrubbing than just doubling or tripling it right away. Some more info. These 2 pools are data pools for a large FS. Unfortunately, we have a large percentage of small files, which is a pain for recovery and seemingly also for deep scrubbing. Our OSDs are about 25% used and I had to increase the warning interval already to 2 weeks. With all the warning grace parameters this means that we manage to deep scrub everything about every month. I need to plan for 75% utilisation and a 3 months period is a bit far on the risky side. Our data is to a large percentage cold data. Client reads will not do the check for us, we need to combat bit-rot pro-actively. The reasons I'm interested in parameters initiating more scrubs while also converting more scrubs into deep scrubs are, that 1) scrubs seem to complete very fast. I almost never catch a PG in state "scrubbing", I usually only see "deep scrubbing". 2) I suspect the low deep-scrub count is due to a low number of deep-scrubs scheduled and not due to conflicting per-OSD deep scrub reservations. With the OSD count we have and the distribution over 12 servers I would expect at least a peak of 50% OSDs being active in scrubbing instead of the 25% peak I'm seeing now. It ought to be possible to schedule more PGs for deep scrub than actually are. 3) Every OSD having only 1 deep scrub active seems to have no measurable impact on user IO. If I could just get more PGs scheduled with 1 deep scrub per OSD it would already help a lot. Once this is working, I can eventually increase osd_max_scrubs when the OSDs fill up. For now I would just like that (deep) scrub scheduling looks a bit harder and schedules more eligible PGs per time unit. If we can get deep scrubbing up to an average of 42PGs completing per hour with keeping osd_max_scrubs=1 to maintain current IO impact, we should be able to complete a deep scrub with 75% full OSDs in about 30 days. This is the current tail-time with 25% utilisation. I believe currently a deep scrub of a PG in these pools takes 2-3 hours. Its just a gut feeling from some repair and deep-scrub commands, I would need to check logs for more precise info. Increasing osd_max_scrubs would then be a further and not the only option to push for more deep scrubbing. My expectation would be that values of 2-3 are fine due to the increasingly higher percentage of cold data for which no interference with client IO will happen. Hope that makes sense and there is a way beyond bumping osd_max_scrubs to increase the number of scheduled and executed deep scrubs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Dan van der Ster <dvanders(a)gmail.com> Sent: 05 January 2023 15:36 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] increasing number of (deep) scrubs Hi Frank, What is your current osd_max_scrubs, and why don't you want to increase it? With 8+2, 8+3 pools each scrub is occupying the scrub slot on 10 or 11 OSDs, so at a minimum it could take 3-4x the amount of time to scrub the data than if those were replicated pools. If you want the scrub to complete in time, you need to increase the amount of scrub slots accordingly. On the other hand, IMHO the 1-week deadline for deep scrubs is often much too ambitious for large clusters -- increasing the scrub intervals is one solution, or I find it simpler to increase mon_warn_pg_not_scrubbed_ratio and mon_warn_pg_not_deep_scrubbed_ratio until you find a ratio that works for your cluster. Of course, all of this can impact detection of bit-rot, which anyway can be covered by client reads if most data is accessed periodically. But if the cluster is mostly idle or objects are generally not read, then it would be preferable to increase slots osd_max_scrubs. Cheers, Dan On Tue, Jan 3, 2023 at 2:30 AM Frank Schilder <frans(a)dtu.dk> wrote: > > Hi all, > > we are using 16T and 18T spinning drives as OSDs and I'm observing that they are not scrubbed as often as I would like. It looks like too few scrubs are scheduled for these large OSDs. My estimate is as follows: we have 852 spinning OSDs backing a 8+2 pool with 2024 and an 8+3 pool with 8192 PGs. On average I see something like 10PGs of pool 1 and 12 PGs of pool 2 (deep) scrubbing. This amounts to only 232 out of 852 OSDs scrubbing and seems to be due to a conservative rate of (deep) scrubs being scheduled. The PGs (dep) scrub fairly quickly. > > I would like to increase gently the number of scrubs scheduled for these drives and *not* the number of scrubs per OSD. I'm looking at parameters like: > > osd_scrub_backoff_ratio > osd_deep_scrub_randomize_ratio > > I'm wondering if lowering osd_scrub_backoff_ratio to 0.5 and, maybe, increasing osd_deep_scrub_randomize_ratio to 0.2 would have the desired effect? Are there other parameters to look at that allow gradual changes in the number of scrubs going on? > > Thanks a lot for your help! > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Re: Status of Quincy 17.2.5 ?
by Christian Rohmann 30 Oct '23

30 Oct '23

Hey everyone, On 20/10/2022 10:12, Christian Rohmann wrote: > 1) May I bring up again my remarks about the timing: > > On 19/10/2022 11:46, Christian Rohmann wrote: > >> I believe the upload of a new release to the repo prior to the >> announcement happens quite regularly - it might just be due to the >> technical process of releasing. >> But I agree it would be nice to have a more "bit flip" approach to >> new releases in the repo and not have the packages appear as updates >> prior to the announcement and final release and update notes. > By my observations sometimes there are packages available on the > download servers via the "last stable" folders such as > https://download.ceph.com/debian-quincy/ quite some time before the > announcement of a release is out. > I know it's hard to time this right with mirrors requiring some time > to sync files, but would be nice to not see the packages or have > people install them before there are the release notes and potential > pointers to changes out. Todays 16.2.11 release shows the exact issue I described above .... 1) 16.2.11 packages are already available via e.g. https://download.ceph.com/debian-pacific 2) release notes not yet merged: (https://github.com/ceph/ceph/pull/49839) thus https://ceph.io/en/news/blog/2022/v16-2-11-pacific-released/ show a 404 :-) 3) No announcement like https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/QOCU563UD3… to the ML yet. Regards Christian

2 2

libceph: mds1 IP+PORT wrong peer at address
by Frank Schilder 20 Sep '23

20 Sep '23

Hi all, we seem to have hit a bug in the ceph fs kernel client and I just want to confirm what action to take. We get the error "wrong peer at address" in dmesg and some jobs on that server seem to get stuck in fs access; log extract below. I found these 2 tracker items https://tracker.ceph.com/issues/23883 and https://tracker.ceph.com/issues/41519, which don't seem to have fixes. My questions: - Is this harmless or does it indicate invalid/corrupted client cache entries? - How to resolve, ignore, umount+mount or reboot? Here an extract from the dmesg log, the error has survived a couple of MDS restarts already: [Mon Mar 6 12:56:46 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:05:18 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:05:18 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:13:50 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-1572619386 [Mon Mar 6 13:13:50 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:41 2023] libceph: mds1 192.168.32.87:6801 socket closed (con state OPEN) [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:45 2023] ceph: mds1 reconnect start [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:16:48 2023] ceph: mds1 reconnect success [Mon Mar 6 13:18:13 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:17 2023] libceph: mds7 192.168.32.88:6801 socket closed (con state OPEN) [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:23 2023] ceph: mds1 recovery completed [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect start [Mon Mar 6 13:18:28 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:29 2023] ceph: mds7 reconnect success [Mon Mar 6 13:18:35 2023] ceph: update_snap_trace error -22 [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:18:35 2023] ceph: mds7 recovery completed [Mon Mar 6 13:22:22 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Mon Mar 6 13:22:22 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Mon Mar 6 13:30:54 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [...] [Thu Mar 9 09:37:24 2023] slurm.epilog.cl (31457): drop_caches: 3 [Thu Mar 9 09:38:26 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:38:26 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:46:58 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:46:58 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 09:55:30 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 09:55:30 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address [Thu Mar 9 10:04:02 2023] libceph: wrong peer, want 192.168.32.87:6801/-223958753, got 192.168.32.87:6801/-453143347 [Thu Mar 9 10:04:02 2023] libceph: mds1 192.168.32.87:6801 wrong peer at address Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 6

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

Re: Ceph Mgr/Dashboard Python depedencies: a new approach
by Casey Bodley 14 Jul '23

14 Jul '23

hi Ernesto and lists, > [1] https://github.com/ceph/ceph/pull/47501 are we planning to backport this to quincy so we can support centos 9 there? enabling that upgrade path on centos 9 was one of the conditions for dropping centos 8 support in reef, which i'm still keen to do if not, can we find another resolution to https://tracker.ceph.com/issues/58832? as i understand it, all of those python packages exist in centos 8. do we know why they were dropped for centos 9? have we looked into making those available in epel? (cc Ken and Kaleb) On Fri, Sep 2, 2022 at 12:01 PM Ernesto Puerta <epuertat(a)redhat.com> wrote: > > Hi Kevin, > >> >> Isn't this one of the reasons containers were pushed, so that the packaging isn't as big a deal? > > > Yes, but the Ceph community has a strong commitment to provide distro packages for those users who are not interested in moving to containers. > >> Is it the continued push to support lots of distros without using containers that is the problem? > > > If not a problem, it definitely makes it more challenging. Compiled components often sort this out by statically linking deps whose packages are not widely available in distros. The approach we're proposing here would be the closest equivalent to static linking for interpreted code (bundling). > > Thanks for sharing your questions! > > Kind regards, > Ernesto > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

4 10

cephadm upgrade 16.2.10 to 16.2.11: osds crash and get stuck restarting
by Zakhar Kirpichenko 14 Jul '23

14 Jul '23

Hi, Attempted to upgrade 16.2.10 to 16.2.11, 2 OSDs out of many started crashing in a loop on the very 1st host: Jan 25 23:07:53 ceph01 bash[2553123]: Uptime(secs): 0.0 total, 0.0 interval Jan 25 23:07:53 ceph01 bash[2553123]: Flush(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(GB): cumulative 0.000, interval 0.000 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Total Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(L0 Files): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: AddFile(Keys): cumulative 0, interval 0 Jan 25 23:07:53 ceph01 bash[2553123]: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Jan 25 23:07:53 ceph01 bash[2553123]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count Jan 25 23:07:53 ceph01 bash[2553123]: ** File Read Latency Histogram By Level [P] ** Jan 25 23:07:53 ceph01 bash[2553123]: debug -10> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986439) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -9> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986493) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -8> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986500) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -7> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: (Original Log Time 2023/01/25-23:07:52.986505) [db_impl/db_impl_compaction_flush.cc:2611] Com paction nothing to do Jan 25 23:07:53 ceph01 bash[2553123]: debug -6> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1676] [O-2] [JOB 9] Compacting 4@0 + 2@1 files to L1, score 1.0 0 Jan 25 23:07:53 ceph01 bash[2553123]: debug -5> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: [compaction/compaction_job.cc:1680] [O-2] Compaction start summary: Base version 33 Base level 0, inputs: [649058(959KB) 649046(1510KB) 649024(1323KB) 649002(1396KB)], [648981(66MB) 648982(52MB)] Jan 25 23:07:53 ceph01 bash[2553123]: debug -4> 2023-01-25T23:07:52.982+0000 7f7c87e67700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1674688072986547, "job": 9, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [649058, 649046, 649024, 649002], "files_L1": [648981, 648982], "score": 1, "input_data_size": 129161327} Jan 25 23:07:53 ceph01 bash[2553123]: debug -3> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 bdev(0x5619bf0ce400 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-1 ((1) Operation not permitted) Jan 25 23:07:53 ceph01 bash[2553123]: debug -2> 2023-01-25T23:07:52.990+0000 7f7c8b66e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f7c8b66e700 time 2023-01-25T23:07:52.993976+0000 Jan 25 23:07:53 ceph01 bash[2553123]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.11/rpm/el8/BUILD/ceph-16.2.11/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!") Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5619b2fc1adc] Jan 25 23:07:53 ceph01 bash[2553123]: 2: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] Jan 25 23:07:53 ceph01 bash[2553123]: 5: clone() Jan 25 23:07:53 ceph01 bash[2553123]: debug -1> 2023-01-25T23:07:52.994+0000 7f7c8b66e700 -1 *** Caught signal (Aborted) ** Jan 25 23:07:53 ceph01 bash[2553123]: in thread 7f7c8b66e700 thread_name:bstore_aio Jan 25 23:07:53 ceph01 bash[2553123]: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) Jan 25 23:07:53 ceph01 bash[2553123]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7c9788fcf0] Jan 25 23:07:53 ceph01 bash[2553123]: 2: gsignal() Jan 25 23:07:53 ceph01 bash[2553123]: 3: abort() Jan 25 23:07:53 ceph01 bash[2553123]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5619b2fc1bad] Jan 25 23:07:53 ceph01 bash[2553123]: 5: (KernelDevice::_aio_thread()+0x1285) [0x5619b3b2a4e5] Jan 25 23:07:53 ceph01 bash[2553123]: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5619b3b357b1] Jan 25 23:07:53 ceph01 bash[2553123]: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f7c978851ca] OSD kept crashing until the host reboot, an OSD restart wouldn't help. This hasn't happened during any previous upgrades, so was a rather unexpected development. Unclear what caused this, but a host reboot seems to have fixed it. It happened to 1 other OSD on another host, exactly the same symptoms, also solved by a reboot. Best regards, Zakhar

4 4

upgrading from 15.2.17 to 16.2.11 - Health ERROR
by xadhoom76＠gmail.com 04 Jul '23

04 Jul '23

hi , starting upgrade from 15.2.17 i got this error Module 'cephadm' has failed: Expecting value: line 1 column 1 (char 0) Cluster was in health ok before starting.

5 9

RadosGW S3 API Multi-Tenancy
by Brad House 08 Jun '23

08 Jun '23

I'm in the process of exploring if it is worthwhile to add RadosGW to our existing ceph cluster. We've had a few internal requests for exposing the S3 API for some of our business units, right now we just use the ceph cluster for VM disk image storage via RBD. Everything looks pretty straight forward until we hit multitenancy. The page on multi-tenancy doesn't dive into permission delegation: https://docs.ceph.com/en/quincy/radosgw/multitenancy/ The end goal I want is to be able to create a single user per tenant (Business Unit) which will act as their 'administrator', where they can then do basically whatever they want under their tenant sandbox (though I don't think we need more advanced cases like creations of roles or policies, just create/delete their own users, buckets, objects). I was hopeful this would just work, and I asked on the ceph IRC channel on OFTC and was told once I grant a user caps="users=*", they would then be allowed to create users *outside* of their own tenant using the Rados Admin API and that I should explore IAM roles. I think it would make sense to add a feature, such as a flag that can be set on a user, to ensure they stay in their "sandbox". I'd assume this is probably a common use-case. Anyhow, if its possible to do today using iam roles/policies, then great, unfortunately this is my first time looking at this stuff and there are some things not immediately obvious. I saw this online about AWS itself and creating a permissions boundary, but that's for allowing creation of roles within a boundary: https://www.qloudx.com/delegate-aws-iam-user-and-role-creation-without-givi… I'm not sure what "Action" is associated with the Rados Admin API create user for applying a boundary that the user can only create users with the same tenant name. https://docs.ceph.com/en/quincy/radosgw/adminops/#create-user Any guidance on this would be extremely helpful. Thanks! -Brad

1 1

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

ln: failed to create hard link 'file name': Read-only file system
by Frank Schilder 26 May '23

26 May '23

Hi all, on an NFS re-export of a ceph-fs (kernel client) I observe a very strange error. I'm un-taring a larger package (1.2G) and after some time I get these errors: ln: failed to create hard link 'file name': Read-only file system The strange thing is that this seems only temporary. When I used "ln src dst" for manual testing, the command failed as above. However, after that I tried "ln -v src dst" and this command created the hard link with exactly the same path arguments. During the period when the error occurs, I can't see any FS in read-only mode, neither on the NFS client nor the NFS server. Funny thing is that file creation and write still works, its only the hard-link creation that fails. For details, the set-up is: file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to other server other server: mount /shares/path as NFS More precisely, on the file-server: fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev 0 0 exports: /shares/nfs/folder -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP On the host at DEST-IP: fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev 0 0 Both, the file server and the client server are virtual machines. The file server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64). When I change the NFS export from "async" to "sync" everything works. However, that's a rather bad workaround and not a solution. Although this looks like an NFS issue, I'm afraid it is a problem with hard links and ceph-fs. It looks like a race with scheduling and executing operations on the ceph-fs kernel mount. Has anyone seen something like that? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 13

Radosgw multisite replication issues
by Tarrago, Eli (RIS-BCT) 12 May '23

12 May '23

Good Afternoon, I am experiencing an issue where east-1 is no longer able to replicate from west-1, however, after a realm pull, west-1 is now able to replicate from east-1. In other words: West <- Can Replicate <- East West -> Cannot Replicate -> East After confirming the access and secret keys are identical on both sides, I restarted all radosgw services. Here is the current status of the cluster below. Thank you for your help, Eli Tarrago root@east01:~# radosgw-admin zone get { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "domain_root": "rgw-east.rgw.meta:root", "control_pool": "rgw-east.rgw.control", "gc_pool": "rgw-east.rgw.log:gc", "lc_pool": "rgw-east.rgw.log:lc", "log_pool": "rgw-east.rgw.log", "intent_log_pool": "rgw-east.rgw.log:intent", "usage_log_pool": "rgw-east.rgw.log:usage", "roles_pool": "rgw-east.rgw.meta:roles", "reshard_pool": "rgw-east.rgw.log:reshard", "user_keys_pool": "rgw-east.rgw.meta:users.keys", "user_email_pool": "rgw-east.rgw.meta:users.email", "user_swift_pool": "rgw-east.rgw.meta:users.swift", "user_uid_pool": "rgw-east.rgw.meta:users.uid", "otp_pool": "rgw-east.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-east.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-east.rgw.buckets.data" } }, "data_extra_pool": "rgw-east.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-east.rgw.log:notif" } root@west01:~# radosgw-admin zone get { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "domain_root": "rgw-west.rgw.meta:root", "control_pool": "rgw-west.rgw.control", "gc_pool": "rgw-west.rgw.log:gc", "lc_pool": "rgw-west.rgw.log:lc", "log_pool": "rgw-west.rgw.log", "intent_log_pool": "rgw-west.rgw.log:intent", "usage_log_pool": "rgw-west.rgw.log:usage", "roles_pool": "rgw-west.rgw.meta:roles", "reshard_pool": "rgw-west.rgw.log:reshard", "user_keys_pool": "rgw-west.rgw.meta:users.keys", "user_email_pool": "rgw-west.rgw.meta:users.email", "user_swift_pool": "rgw-west.rgw.meta:users.swift", "user_uid_pool": "rgw-west.rgw.meta:users.uid", "otp_pool": "rgw-west.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-west.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-west.rgw.buckets.data" } }, "data_extra_pool": "rgw-west.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-west.rgw.log:notif" east01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "init", "num_shards": 0, "period": "", "realm_epoch": 0 }, "markers": [] }, "full_sync": { "total": 0, "complete": 0 } } west01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "sync", "num_shards": 64, "period": "44b6b308-e2d8-4835-8518-c90447e7b55c", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 46, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, #### goes on for a long time… { "key": 63, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } } ] }, "full_sync": { "total": 46, "complete": 46 } } east01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) metadata sync no sync (zone is master) 2023-04-20T19:03:13.388+0000 7f25fa036c80 0 ERROR: failed to fetch datalog info data sync source: b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) failed to retrieve sync info: (13) Permission denied west01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 16 shards behind shards: [5,56,62,65,66,70,76,86,87,94,104,107,111,113,120,126] oldest incremental change not applied: 2023-04-20T19:02:48.783283+0000 [5] east01:~# radosgw-admin zonegroup get { "id": "0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd", "name": "EastWestceph", "api_name": "EastWestceph", "is_master": "true", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080, http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "hostnames": [ "eastvip.example.net", "westvip.example.net" ], "hostnames_s3website": [], "master_zone": "ddd66ab8-0417-46ee-a53b-043352a63f93", "zones": [ { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "endpoints": [ http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "sync_policy": { "groups": [] } } ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

4 6

Lua scripting in the rados gateway
by Thomas Bennett 11 May '23

11 May '23

Hi ceph users, I've been trying out the lua scripting for the rados gateway (thanks Yuval). As in my previous email I mentioned that there is an error when trying to load the luasocket module. However, I thought it was a good time to report on my progress. My 'hello world' example below is called *test.lua* below includes the following checks: 1. Can I write to the debug log? 2. Can I use the lua socket package to do something stupid but intersting, like connect to a webservice? Before you continue reading this, you might need to know that I run all ceph processes in a *CentOS Stream release 8 *container deployed using ceph orchestrator running *Ceph v17.2.5*, so please view the information below in that context. For anyone looking for a reference, I suggest going to the ceph lua rados gateway documentation at radosgw/lua-scripting <https://docs.ceph.com/en/quincy/radosgw/lua-scripting/>. There are two new switches you need to know about in the radosgw-admin: - *script* -> loading your lua script - *script-package* -> loading supporting packages for your script - e.i. luasocket in this case. For a basic setup, you'll need to have a few dependencies in your containers: - cephadm container: requires luarocks (I've checked the code - it runs a luarocks search command) - radosgw container: requires luarocks, gcc, make, m4, wget (wget just in case). To achieve the above, I updated the container image for our running system. I needed to do this because I needed to redeploy the rados gateway container to inject the lua script packages into the radosgw runtime process. This will start with a fresh container based on the global config *container_image* setting on your running system. For us this is currently captured in *quay.io/tsolo/ceph:v17.2.5-3 <http://quay.io/tsolo/ceph:v17.2.5-3>* and included the following exta steps (including installing the lua dev from an rpm because there is no centos package in yum): yum install luarocks gcc make wget m4 rpm -i https://rpmfind.net/linux/centos/8-stream/PowerTools/x86_64/os/Packages/lua… You will notice that I've included a compiler and compiler support into the image. This is because luarocks on the radosgw to compile luasocket (the package I want to install). This will happen at start time when the radosgw is restarted from ceph orch. In the cephadm container I still need to update our cephadm shell so I need to install luarocks by hand: yum install luarocks Then set thew updated image to use: ceph config set global container_image quay.io/tsolo/ceph:v17.2.5-3 I now create a file called: *test.lua* in the cephadm container. This contains the following lines to write to the log and then do a get request to google. This is not practical in production, but it serves the purpose of testing the infrastructure: RGWDebugLog("Tsolo start lua script") local LuaSocket = require("socket") client = LuaSocket.connect("google.com", 80) client:send("GET / HTTP/1.0\r\nHost: google.com\r\n\r\n") while true do s, status, partial = client:receive('*a') RGWDebugLog(s or partial) if status == "closed" then break end end client:close() RGWDebugLog("Tsolo stop lua") Next I run: radosgw-admin script-package add --package=luasocket --allow-compilation And then list the added package to make sure it is there: radosgw-admin script-package list Note - at this point the radosgw has not been modified, it must first be restarted. Then I put the *test.lua *script into the pre request context: radosgw-admin script put --infile=test.lua --context=preRequest You also need to raise the debug log level on the running rados gateway: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.asok config set debug_rgw 20 Inside the radosgw container I apply my fix (as per previous email): cp -ru /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib64/* /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib/ Outside on the host running the radosgw-admin container I follow the journalctl for the radosgw container (to get the logs): journalctl -fu ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@rgw. xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.service Then I run an s3cmd to put data in via the rados gateway and check the journalctl logs and see: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua So the script worked :) If you want to see where the luarocks libraries have been installed, look in the /tmp/ directory of the radosgw container after you redeploy it and you will find the content in /tmp/luarocks. Conclusions: There was a bit to figure out to get this working, but now that I've got this simple test working I think there is a lot more to look into and discover and use w.r.t. this powerful tool. Cheers, Tom

2 5

How can I use not-replicated pool (replication 1 or raid-0)
by mhnx 10 May '23

10 May '23

Hello. I have a 10 node cluster. I want to create a non-replicated pool (replication 1) and I want to ask some questions about it: Let me tell you my use case: - I don't care about losing data, - All of my data is JUNK and these junk files are usually between 1KB to 32MB. - These files will be deleted in 5 days. - Writable space and I/O speed is more important. - I have high Write/Read/Delete operations, minimum 200GB a day. I'm afraid that, in any failure, I won't be able to access the whole cluster. Losing data is okay but I have to ignore missing files, remove the data from the cluster and continue with existing data and while doing this, I want to be able to write new data to the cluster. My questions are: 1- To reach this goal do you have any recommendations? 2- With this setup, what potential problems do you have in mind? 3- I think Erasure Coding is not a choice because of the performance problems and slow file deletion. With this I/O need EC will miss files and leaks may happen (I've seen before on Nautilus). 4- You read my needs, is there a better way to do this? Thank you for the answers. Best regards.

3 8

ERROR: Distro uos version 20 not supported
by Ben 09 May '23

09 May '23

Hi, This seems not very relevant since all ceph components are running in containers. Any ideas to get over this issue? Any other ideas or suggestions on this kind of deployment? sudo ./cephadm --image 10.21.22.1:5000/ceph:v17.2.5-20230316 --docker bootstrap --mon-ip 10.21.22.1 --skip-monitoring-stack Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... No time sync service is running; checked for ['chrony.service', 'chronyd.service', 'systemd-timesyncd.service', 'ntpd.service', 'ntp.service', 'ntpsec.service', 'openntpd.service'] ERROR: Distro uos version 20 not supported uname -a Linux aaaa 4.19.0-91.82.42.uelc20.x86_64 #1 SMP Sat May 15 13:50:04 CST 2021 x86_64 x86_64 x86_64 GNU/Linux Thank you in advance Ben

1 1

16.2.13 pacific QE validation status
by Yuri Weinstein 08 May '23

08 May '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59542#note-1 Release Notes - TBD Seeking approvals for: smoke - Radek, Laura rados - Radek, Laura rook - Sébastien Han cephadm - Adam K dashboard - Ernesto rgw - Casey rbd - Ilya krbd - Ilya fs - Venky, Patrick upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8) upgrade/pacific-p2p - Laura powercycle - Brad (SELinux denials) ceph-volume - Guillaume, Adam K Thx YuriW

10 26

Frequent calling monitor election
by Frank Schilder 04 May '23

04 May '23

Hi all, our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting: 2/9/23 4:42:30 PM[INF]overall HEALTH_OK 2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:40:00 PM[INF]overall HEALTH_OK 2/9/23 4:30:00 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:04 PM[INF]overall HEALTH_OK 2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:20:00 PM[INF]overall HEALTH_OK 2/9/23 4:10:00 PM[INF]overall HEALTH_OK 2/9/23 4:00:00 PM[INF]overall HEALTH_OK 2/9/23 3:50:00 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again? Thanks for any help! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 9

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

RBD mirroring, asking for clarification
by wodel youchi 03 May '23

03 May '23

Hi, When using rbd mirroring, the mirroring concerns the images only, not the whole pool? So, we don't need to have a dedicated pool in the destination site to be mirrored, the only obligation is that the mirrored pools must have the same name. In other words, We create two pools with the same name, one on the source site the other on the destination site, we create the mirror link (one way or two ways replication), then we choose what images to sync. Both pools can be used simultaneously on both sites, it's the mirrored images that cannot be used simultaneously, only promoted ones. Is this correct? Regards.

2 5

architecture help (iscsi, rbd, backups?)
by Angelo Höngens 02 May '23

02 May '23

Hey guys and girls, I'm working on a project to build storage for one of our departments, and I want to ask you guys and girls for input on the high-level overview part. It's a long one, I hope you read along and comment. SUMMARY I made a plan last year to build a 'storage solution' including ceph and some windows VM's to expose the data over SMB to clients. A year later I finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph itself runs great, but when I wanted to start exposing the data using iscsi to our VMware farm, I ran into some issues. I know the iscsi gateways will introduce some new performance bottlenecks, but I'm seeing really slow performance, still working on that. But then I ran into the warning on the iscsi gateway page: "The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features.". Wait, what? Why!? What does this mean? Does this mean that iSCSI is now 'feature complete' and will still be supported the next 5 years, or will it be deprecated in the future? I tried searching, but couldn't find any info on the decision and the roadmap. My goal is to build a future-proof setup, and using deprecated components should not be part of that of course. If the iscsi gateway will still be supported the next few years and I can iron out the performance issues, I can still go on with my original plan. If not, I have to go back to the drawing board. And maybe you guys would advise me to take another route anyway. GOALS My goals/considerations are: - we want >1PB of storage capacity for cheap (on a tight budget) for research data. Most of it is 'store once, read sometimes'. <1% of the data is 'hot'. - focus is on capacity, but it would be nice to have > 200MB/s of sequential write/read performance and not 'totally suck' on random i/o. Yes, not very well quantified, but ah. Sequential writes are most important. - end users all run Windows computers (mostly VDI's) and a lot of applications require SMB shares. - security is a big thing, we want really tight ACL's, specific monitoring agents, etc. - our data is incredibly important to us, we still want the 3-2-1 backup rule. Primary storage solution, a second storage solution in a different place, and some of the data that is not reproducible is also written to tape. We also want to be protected from ransomware or user errors (so no direct replication to the second storage). - I like open source, reliability, no fork-lift upgrades, no vendor lock-in, blah, well, I'm on the ceph list here, no need to convince you guys ;) - We're hiring a commercial company to do ceph maintenance and support for when I'm on leave or leaving the company, but they won't support clients, backup software, etc, so I want something as simple as possible. We do have multiple Windows/VMware admins, but no other real linux guru's. THE INITIAL PLAN Given these considerations, I ordered two identical clusters, each consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and 10 capacity disks (EC 4:2 for the data), and each node is connected using a 2x25Gbps bond. Ceph is running like a charm. Now I just have to think about exposing the data to end users, and I've been testing different setups. My original plan was to expose for example 10x100TB rbd images using iSCSI to our VMware farm, formatting the luns with VMFS6, and run for example 2 Windows file servers per datastore on that with a single DFS namespace to end users. Then backup the file servers using our existing Veeam infrastructure to RGW running on the second cluster with an immutable bucket. This way we would have easily defined security boundaries: the clients can only reach the file servers, the file servers only see their local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file server would be compromised, it would have no access to ceph. We have easy incremental backups, immutability for ransomware protection, etc. And the best part is that the ceph admin can worry about ceph, the vmware admin can focus on ESX, VMFS and all the vmware stuff, and the Windows admins can focus on the Windows boxes, Windows-specific ACLS and tools and Veeam backups and stuff. CURRENT SITUATION I'm building out this plan now, but I'm running into issues with iSCSI. Are any of you doing something similar? What is your iscsi performance compared to direct rbd? In regard to performance: If I take 2 test windows VM's, I put one on an iSCSI datastore and another with direct rbd access using the windows rbd driver, I create a share on those boxes and push data to it, I see different results (of course). Copying some iso images over SMB to the 'windows vm running direct rbd' I see around 800MB/s write, and 200MB/s read, which is pretty okay. When I send data to the 'windows vm running on top of iscsi' it starts writing at around 350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really acceptable. Another really viable and performant scenario would be to have the Windows file servers connect to rbd directly with the windows rbd driver. It seems to work well, it's fast, and you don't have the bottleneck that the iscsi gateway creates. But I see this driver is still in beta. Is anyone using this in production? What are your experiences? We would miss out on the separation of layers and thus have less security, but at the same time, it really increases efficiency and performance. And if I use rbd, then vmware won't see the storage, and I cannot do an image backup using veeam. I could of course do backups of the rbd images, using tools like restic or backy to rgw running on the second cluster with immutable buckets. What are your experiences? Is it easy to do differential backups of lots of 50TB rbd images? Change rate is usually like 0.005% per day or something ;) By the way, we also thought about CephFS, but we have some complex stuff going on with extended ACL's that I don't think will play nice with CephFS, and I think it's a lot more complex to backup CephFS than block images. If you made it here, thank you for your time! I hope you can share thoughts on my questions! Angelo.

6 8

Memory leak in MGR after upgrading to pacific.
by Gary Molenkamp 02 May '23

02 May '23

Good morning, After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days ago, I'm noticing that the MGR daemons keep failing over to standby and then back every 24hrs. Watching the output of 'ceph orch ps' I can see that the memory consumption of the mgr is steadily growing until it becomes unresponsive. When the mgr becomes unresponsive, tasks such as RESTful calls start to fail, and the standby eventually takes over after ~20 minutes. I've included a log of memory consumption (in 10 minute intervals) at the end of this message. While the cluster recovers during this issue, the loss of usage data during the outage, and the fact its occurring is problematic. Any assistance would be appreciated. Note, this is a cluster that has been upgraded from an original jewel based ceph using filestore, through bluestore conversion, container conversion, and now to Pacific. The data below shows memory use with three mgr modules enabled: cephadm, restful, iostat. By disabling iostat, I can reduce the rate of memory consumption increasing to about 200MB/hr. Thanks Gary. Wed Apr 26 12:50:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 7m ago 3h 1099M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 7m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:00:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1161M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:10:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1230M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:20:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 6m ago 4h 1250M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:30:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1318M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 5m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:40:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:50:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1442M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:00:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1498M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:10:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1554M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:20:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 4m ago 5h 1617M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:30:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1677M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:40:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1735M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 3m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:50:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1806M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:00:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:10:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1920M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:20:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 114s ago 6h 1981M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 113s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:30:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 95s ago 6h 2048M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 95s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:40:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 80s ago 6h 2121M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 80s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:50:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 53s ago 6h 2170M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 53s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:00:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 35s ago 6h 2238M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 34s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:10:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 19s ago 6h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 19s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:20:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 10m ago 7h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 10m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:30:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2321M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 9m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:40:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2382M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:50:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2449M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:00:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2505M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:10:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2562M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:20:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 8m ago 8h 2628M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:30:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2692M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 7m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:40:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2761M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:50:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2811M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:00:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:10:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2923M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:20:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 6m ago 9h 2995M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:30:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3055M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 5m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:40:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3110M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:50:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3166M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:00:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3244M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:10:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3289M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:20:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 4m ago 10h 3368M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:30:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3437M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 3m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:40:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3496M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:50:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3546M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:00:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3627M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:10:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3637M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:20:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 92s ago 11h 492M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 91s ago 23h 418M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:31:55 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 51s ago 11h 373M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 51s ago 23h 520M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:41:56 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 32s ago 11h 374M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 32s ago 0h 585M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:51:57 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 17s ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 17s ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:01:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 10m ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 10m ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:11:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 9m ago 11h 378M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 698M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:21:59 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 758M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:32:00 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 825M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:42:01 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 895M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:52:02 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 948M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:02:03 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 1010M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:12:04 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 7m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1071M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:22:05 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1132M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:32:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1183M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:42:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1244M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:52:07 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1311M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:02:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1380M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:12:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 5m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1441M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:22:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1508M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:32:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1541M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:42:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1610M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:52:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1674M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:02:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1725M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:12:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 3m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1794M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:22:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1844M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:32:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1917M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:42:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2002M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:52:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2056M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:02:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2113M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:12:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 92s ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 91s ago 4h 2173M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:22:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 73s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 72s ago 4h 2201M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:32:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 57s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 57s ago 4h 2264M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:42:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 29s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 29s ago 5h 2324M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:52:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10s ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:02:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10m ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10m ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:12:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 9m ago 16h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2442M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:22:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2502M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:32:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2552M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:42:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2614M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:52:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2681M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:02:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2748M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:12:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 7m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2810M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:22:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2867M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:32:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2916M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:42:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3024M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:52:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3070M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:02:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3127M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:12:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 5m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3183M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:22:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 5m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3215M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:32:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 4m ago 7h 3290M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:42:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3342M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:52:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3392M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:02:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3451M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:12:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3516M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:22:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 3m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3566M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:32:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 2m ago 8h 3632M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:42:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3696M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:52:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3760M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:02:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 115s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 114s ago 9h 3813M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:12:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 99s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 99s ago 9h 3869M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:22:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 72s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 72s ago 9h 3923M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:32:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 54s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 53s ago 9h 3990M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:42:49 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 38s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 38s ago 10h 4040M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:52:50 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 11s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10s ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:02:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 10m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10m ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:12:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 9m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4157M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:22:52 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (22h) 9m ago 22h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4213M - 16.2.12 921c4e969fff 6fed006895e8 -- Gary Molenkamp Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

1 1

PVE CEPH OSD heartbeat show
by Peter 02 May '23

02 May '23

Dear all, We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues. Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU. When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes. We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline. We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,

3 4

Re: cephfs - max snapshot limit?
by MARTEL Arnaud 02 May '23

02 May '23

Hi Venky, > Also, at one point the kclient wasn't able to handle more than 400 snapshots (per file system), but we have come a long way from that and that is not a constraint right now. Does it mean that there is no more limit to the number of snapshots per filesystem? And, if not, do you know what is the max number of snapshots now (per filesystem) ?? Cheers, Arnaud

2 1

Ceph recovery
by wodel youchi 01 May '23

01 May '23

Hi, When creating a ceph cluster, a failover domain is created, and by default it uses host as a minimal domain, that domain can be modified to chassis, or rack, ...etc. My question is : Suppose I have three osd nodes, my replication is 3 and my failover domain is host, which means that each copy of data is stored on a different node. What happens when one node crashes, does Ceph use the remaining free space on the other two to create the third copy, or the ceph cluster will run in degraded mode, like a RAID5 which lost a disk. Regards.

2 2

Nearly 1 exabyte of Ceph storage
by Yaarit Hatuka 01 May '23

01 May '23

Hi everyone, We are excited to share with you the latest statistics from our Ceph public telemetry dashboards <https://telemetry-public.ceph.com/>. With almost 2,500 clusters reporting nearly 1 exabyte of total storage capacity, we can see a steady growth rate <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=19&orgId=…> since the telemetry service was launched. One of the things telemetry helps us to understand is version adoption rate. See, for example, the trend of Quincy <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=28&orgId=…> deployments in the community. We can also learn that the average cluster size grows over time <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=33&orgId=1>. Ceph telemetry is on an opt-in basis. You can opt-in with: `ceph telemetry on` Learn more here <https://docs.ceph.com/en/latest/mgr/telemetry/#enabling-telemetry>. Help us cross the exabyte mark by opting-in today! Learn more about the latest developments around Telemetry <https://sched.co/1JKZ2> at the upcoming Cephalocon. Thank you for your contribution, Yaarit [image: ceph_telemetry_dashboards.jpg]

2 3

RGW Lua - cancel request
by Ondřej Kukla 01 May '23

01 May '23

Hello, Lately I’ve been playing with Lua scripting on top of RGW. I would like to implement a request blocking based on bucket name -> when there is a dot in a bucket name return error code and a message that this name is invalid. Here is the code I was able to came up with. if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = “InvalidBucketName" Request.Response.Message = “Dots in bucket name are not allowed." end This works fine, but the request for creating a bucket would be processed and the bucket will be created. I thought about a dirty workaround with setting the Request.Bucket.Name to a bucket that already exists but it seems that this field is not writable in Quincy. Is there a way to block the request from processing? Any help is much appreciated. Kind regards, Ondrej

3 3

backing up CephFS
by Angelo Höngens 30 Apr '23

30 Apr '23

How do you guys backup CephFS? (if at all?) I'm building 2 ceph clusters, a primary one and a backup one, and I'm looking into CephFS as the primary store for research files. CephFS mirroring seems a very fast and efficient way to copy data to the backup location, and it has the benefit of the files on the backup location being fully in a ready-to-use state instead of some binary proprietary archive. But I am wondering how to do 'ransomware protection' in this setup. I can't believe I'm the only one that wants to secure my data ;) I'm reading up on snapshots and mirroring, and that's great to protect from user error. I could schedule snapshots on the primary cluster, and they would automatically get synced to the backup cluster. But a user can still delete all snapshots on the source side, right? And you need to create a ceph user on the backup cluster, and import that on the primary cluster. That means that if a hacker has those credentials, he could also delete the data on the backup cluster? Or is there some 'append-only' mode for immutability? Another option I'm looking into is restic. Restic looks like a cool tool, but it does not support s3 object locks yet. See the discussion here [1]. I should be able to get immutability working with the restic-rest backend according to the developer. But I have my worries that running restic to sync up an 800TB filesystem with millions of files will be.. worrysome ;) Anyone using restic in production? Thanks again for your input! Angelo. [1] https://github.com/restic/restic/issues/3195

4 3

Deep-scrub much slower than HDD speed
by Niklas Hambüchen 30 Apr '23

30 Apr '23

I observed that on an otherwise idle cluster, scrubbing cannot fully utilise the speed of my HDDs. `iostat` shows only 8-10 MB/s per disk, instead of the ~100 MB/s most HDDs can easily deliver. Changing scrubbing settings does not help (see below). Environment: * 6 active+clean+scrubbing+deep * Ceph version 16.2.7. * BlueStore * My cluster has many objects small objects ("402.32M objects, 38 TiB" from "ceph status") due to small files (4 - 32 KiB) on CephFS. `iostat -x 5` with default settings: Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 198.60 6878.40 0.00 0.00 12.78 34.63 51.80 2612.80 0.00 0.00 14.82 50.44 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.30 91.24 dm-1 0.80 3.20 0.00 0.00 11.50 4.00 52.60 2582.40 0.00 0.00 13.69 49.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.73 3.78 dm-10 11.20 71.20 0.00 0.00 0.09 6.36 145.80 583.20 0.00 0.00 0.14 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 2.62 dm-11 192.60 6737.60 0.00 0.00 10.74 34.98 34.80 1684.80 0.00 0.00 11.47 48.41 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.47 91.40 dm-12 245.40 10194.40 0.00 0.00 9.43 41.54 21.20 575.20 0.00 0.00 3.94 27.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.40 87.92 dm-13 30.80 1772.80 0.00 0.00 11.61 57.56 78.80 4507.20 0.00 0.00 19.78 57.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.92 9.54 dm-14 3.20 24.80 0.00 0.00 0.12 7.75 125.20 500.80 0.00 0.00 0.12 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.18 dm-15 2.80 19.20 0.00 0.00 0.14 6.86 105.40 421.60 0.00 0.00 0.05 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.76 dm-16 0.80 6.40 0.00 0.00 0.00 8.00 111.00 444.00 0.00 0.00 0.10 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.82 dm-17 10.80 76.80 0.00 0.00 0.09 7.11 151.40 605.60 0.00 0.00 0.08 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.92 dm-18 10.20 67.20 0.00 0.00 0.08 6.59 115.60 462.40 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.16 dm-19 10.20 56.80 0.00 0.00 0.10 5.57 109.00 436.00 0.00 0.00 0.07 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.34 dm-2 4.80 435.20 0.00 0.00 0.12 90.67 751.80 6292.80 0.00 0.00 0.07 8.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 16.14 dm-20 0.40 2.40 0.00 0.00 0.00 6.00 265.00 2459.20 0.00 0.00 0.10 9.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 5.36 dm-21 191.00 6105.60 0.00 0.00 6.34 31.97 67.80 3748.00 0.00 0.00 19.56 55.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.54 42.22 dm-3 1.00 8.80 0.00 0.00 0.00 8.80 91.00 364.00 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.54 dm-4 167.60 4973.60 0.00 0.00 10.15 29.68 49.20 2511.20 0.00 0.00 11.39 51.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.26 89.18 dm-5 11.20 73.60 0.00 0.00 0.12 6.57 124.40 497.60 0.00 0.00 0.07 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.16 dm-6 27.20 1644.80 0.00 0.00 12.22 60.47 57.20 3316.80 0.00 0.00 15.93 57.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.24 6.78 dm-7 217.40 8032.80 0.00 0.00 12.04 36.95 64.40 3654.40 0.00 0.00 23.69 56.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.14 97.28 dm-8 10.80 70.40 0.00 0.00 0.15 6.52 111.80 447.20 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.32 dm-9 1.60 8.00 0.00 0.00 13.25 5.00 46.60 2563.20 0.00 0.00 9.08 55.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.44 3.70 md127 1.60 107.20 0.00 0.00 0.00 67.00 142.00 1856.00 0.00 0.00 0.05 13.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.80 nvme0n1 42.00 772.00 0.00 0.00 0.10 18.38 1001.00 10503.50 451.20 31.07 0.03 10.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 25.20 nvme1n1 37.20 248.00 0.00 0.00 0.09 6.67 609.80 6723.50 369.00 37.70 0.04 11.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 15.80 sda 0.80 3.20 0.00 0.00 11.50 4.00 33.60 2582.40 19.00 36.12 8.43 76.86 0.00 0.00 0.00 0.00 0.00 0.00 3.20 0.25 0.29 3.78 sdb 117.80 10195.20 127.80 52.04 11.53 86.55 18.80 575.20 2.40 11.32 3.93 30.60 0.00 0.00 0.00 0.00 0.00 0.00 2.80 3.93 1.44 87.90 sdc 4.60 1644.80 22.60 83.09 11.74 357.57 23.60 3316.80 33.60 58.74 8.53 140.54 0.00 0.00 0.00 0.00 0.00 0.00 1.80 4.56 0.26 6.74 sdd 109.40 4975.20 58.40 34.80 12.19 45.48 22.00 2511.20 27.20 55.28 5.75 114.15 0.00 0.00 0.00 0.00 0.00 0.00 2.80 6.07 1.48 89.16 sde 115.40 6563.20 77.00 40.02 12.21 56.87 26.20 1684.80 8.60 24.71 7.82 64.31 0.00 0.00 0.00 0.00 0.00 0.00 1.80 6.78 1.63 91.36 sdf 6.40 1772.80 24.40 79.22 14.84 277.00 39.20 4507.20 39.60 50.25 13.93 114.98 0.00 0.00 0.00 0.00 0.00 0.00 3.20 2.25 0.65 9.54 sdg 121.60 8033.60 95.80 44.07 12.39 66.07 30.60 3654.40 33.80 52.48 12.47 119.42 0.00 0.00 0.00 0.00 0.00 0.00 2.60 8.77 1.91 97.24 sdh 122.00 6105.60 69.00 36.13 5.47 50.05 33.00 3748.00 34.80 51.33 16.94 113.58 0.00 0.00 0.00 0.00 0.00 0.00 2.20 5.00 1.24 42.20 sdi 117.00 6856.80 81.60 41.09 12.25 58.61 32.00 2612.80 19.80 38.22 10.80 81.65 0.00 0.00 0.00 0.00 0.00 0.00 2.60 8.46 1.80 91.18 sdj 1.60 8.00 0.00 0.00 13.12 5.00 31.20 2563.20 15.40 33.05 5.06 82.15 0.00 0.00 0.00 0.00 0.00 0.00 2.00 1.70 0.18 3.70 With settings osd_deep_scrub_stride = 4194304 osd_scrub_load_threshold = 20 osd_scrub_chunk_min = 15 osd_scrub_chunk_max = 75 osd_max_scrubs = 3 I get a slight improvement only: Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util dm-0 400.60 14686.40 0.00 0.00 19.93 36.66 25.20 1197.60 0.00 0.00 27.44 47.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8.67 91.32 dm-1 362.60 10583.20 0.00 0.00 18.58 29.19 30.40 1742.40 0.00 0.00 23.32 57.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.45 98.14 dm-10 6.40 64.00 0.00 0.00 0.03 10.00 37.60 150.40 0.00 0.00 0.06 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.02 dm-11 76.60 1939.20 0.00 0.00 10.61 25.32 35.20 1885.60 0.00 0.00 22.32 53.57 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.60 29.06 dm-12 93.00 4178.40 0.00 0.00 16.04 44.93 4.40 96.00 0.00 0.00 0.68 21.82 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.50 19.04 dm-13 376.80 12716.00 0.00 0.00 15.90 33.75 27.20 1444.80 0.00 0.00 23.76 53.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.64 97.76 dm-14 11.80 312.80 0.00 0.00 0.12 26.51 33.40 133.60 0.00 0.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 dm-15 7.00 430.40 0.00 0.00 0.17 61.49 46.20 184.80 0.00 0.00 0.06 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.94 dm-16 11.80 229.60 0.00 0.00 0.10 19.46 49.20 196.80 0.00 0.00 0.09 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.20 dm-17 4.00 39.20 0.00 0.00 0.10 9.80 29.60 118.40 0.00 0.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.72 dm-18 9.00 94.40 0.00 0.00 0.07 10.49 37.20 148.80 0.00 0.00 0.30 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.92 dm-19 7.40 70.40 0.00 0.00 0.11 9.51 56.80 227.20 0.00 0.00 0.02 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.04 dm-2 0.00 0.00 0.00 0.00 0.00 0.00 62.00 482.40 0.00 0.00 0.07 7.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.16 dm-20 12.40 2375.20 0.00 0.00 0.23 191.55 314.20 1524.80 0.00 0.00 0.03 4.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 7.90 dm-21 321.00 9468.00 0.00 0.00 7.86 29.50 31.20 1630.40 0.00 0.00 11.69 52.26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.89 42.84 dm-3 12.40 125.60 0.00 0.00 0.08 10.13 37.00 148.00 0.00 0.00 0.02 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.84 dm-4 374.00 13150.40 0.00 0.00 18.31 35.16 23.60 1100.80 0.00 0.00 12.42 46.64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.14 96.70 dm-5 8.00 84.00 0.00 0.00 0.10 10.50 38.00 152.00 0.00 0.00 0.03 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 dm-6 201.60 6619.20 0.00 0.00 11.56 32.83 3.40 108.80 0.00 0.00 2.29 32.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.34 58.24 dm-7 414.40 14476.00 0.00 0.00 21.99 34.93 10.80 235.20 0.00 0.00 5.19 21.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.17 98.00 dm-8 0.60 14.40 0.00 0.00 0.00 24.00 40.80 163.20 0.00 0.00 0.04 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.70 dm-9 478.00 17300.80 0.00 0.00 23.06 36.19 4.40 92.00 0.00 0.00 5.00 20.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11.05 98.00 md127 5.00 64.80 0.00 0.00 0.04 12.96 133.80 2014.40 0.00 0.00 0.08 15.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.32 nvme0n1 33.60 822.40 0.00 0.00 0.09 24.48 252.40 3388.10 139.40 35.58 0.04 13.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 7.24 nvme1n1 62.60 3082.40 0.00 0.00 0.10 49.24 427.40 4272.90 177.20 29.31 0.03 10.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 13.70 sda 236.20 10613.60 126.20 34.82 15.19 44.93 11.80 1742.40 18.60 61.18 13.83 147.66 0.00 0.00 0.00 0.00 0.00 0.00 1.40 20.29 3.78 98.14 sdb 35.80 4166.40 56.80 61.34 10.03 116.38 4.20 96.00 0.20 4.55 0.81 22.86 0.00 0.00 0.00 0.00 0.00 0.00 0.80 0.25 0.36 19.04 sdc 118.20 6612.80 83.00 41.25 9.15 55.95 3.00 108.80 0.40 11.76 2.07 36.27 0.00 0.00 0.00 0.00 0.00 0.00 0.60 3.67 1.09 58.20 sdd 207.00 13152.00 166.60 44.59 13.92 63.54 15.80 1100.80 7.80 33.05 6.90 69.67 0.00 0.00 0.00 0.00 0.00 0.00 1.80 9.67 3.01 96.66 sde 55.80 1939.20 20.80 27.15 9.06 34.75 17.40 1885.60 17.80 50.57 7.51 108.37 0.00 0.00 0.00 0.00 0.00 0.00 1.80 1.44 0.64 29.04 sdf 221.40 12720.00 155.20 41.21 14.34 57.45 14.60 1444.80 12.60 46.32 10.51 98.96 0.00 0.00 0.00 0.00 0.00 0.00 1.40 11.71 3.35 97.70 sdg 234.60 14490.40 179.80 43.39 15.66 61.77 9.40 235.20 1.20 11.32 5.36 25.02 0.00 0.00 0.00 0.00 0.00 0.00 2.00 11.10 3.75 97.98 sdh 212.20 9470.40 109.20 33.98 4.05 44.63 19.40 1630.40 11.80 37.82 6.41 84.04 0.00 0.00 0.00 0.00 0.00 0.00 2.00 4.10 0.99 42.78 sdi 214.00 14684.00 186.60 46.58 14.46 68.62 10.00 1197.60 15.20 60.32 11.14 119.76 0.00 0.00 0.00 0.00 0.00 0.00 2.40 13.67 3.24 91.28 sdj 250.60 17296.80 227.00 47.53 18.76 69.02 3.60 92.00 0.80 18.18 5.72 25.56 0.00 0.00 0.00 0.00 0.00 0.00 0.80 16.25 4.73 98.00 `osd_max_scrubs` creates more reads per second (~120 -> ~220), but does not proportionately increase read throughput (as is expected on a spinning disk). Overall, this looks like scrub operations are seek-bound. Looking at `strace -fyp 640105 -e io_submit` output, for lines that involve my HDD `/dev/dm-12`: io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd55baa3000, iov_len=4096}], aio_offset=1654046691328}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f378000, iov_len=24576}], aio_offset=1654046666752}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f262000, iov_len=4096}], aio_offset=1934563819520}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f28b000, iov_len=4096}], aio_offset=1934563823616}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3c7000, iov_len=28672}], aio_offset=2871307956224}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f2d2000, iov_len=4096}], aio_offset=2871307984896}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f34a000, iov_len=8192}], aio_offset=4056494669824}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f35f000, iov_len=4096}], aio_offset=4056494678016}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3e3000, iov_len=53248}], aio_offset=1233895051264}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f0c6000, iov_len=4096}], aio_offset=1233895104512}]) = 2 io_submit(0x7fd5f6668000, 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f1e2000, iov_len=4096}], aio_offset=2155792314368}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f303000, iov_len=24576}], aio_offset=2155792289792}]) = 2 This looks like it's reading the HDD all over the place, with small reads (`aio_offset`, `iov_len`). Does Ceph scrubbing do a HDD seek for every object? I had imagined that scrubbing might be able to read linearly through the disk (perhaps skipping larger gaps). Is that wrong? Thanks!

7 16

Enable LUKS encryption on a snapshot created from unencrypted image
by Will Gorman 30 Apr '23

30 Apr '23

Is there a way to enable the LUKS encryption format on a snapshot that was created from an unencrypted image without losing data? I've seen in https://docs.ceph.com/en/quincy/rbd/rbd-encryption/ that "Any data written to the image prior to its format may become unreadable, though it may still occupy storage resources." and observed that to be the case when running `encryption format` on an image that already has data in it. However is there any way to take a snapshot of an unencrypted image and enable encryption on the snapshot (or even on a new image cloned from the snapshot?)

2 1

cephfs - max snapshot limit?
by Tobias Hachmer 29 Apr '23

29 Apr '23

Hello, we are running a 3-node ceph cluster with version 17.2.6. For CephFS snapshots we have configured the following snap schedule with retention: /PATH 2h 72h15d6m But we observed that max 50 snapshot are preserved. If a new snapshot is created the oldest 51st is deleted. Is there a limit for maximum cephfs snapshots or maybe this is a bug? I have found the setting "mds_max_snaps_per_dir" which is 100 by default but I think this is not related to my problem? Thanks, Tobias

4 9

Return code -116 when starting MDS scrub
by Thomas Widhalm 29 Apr '23

29 Apr '23

Hi, I followed the steps to repair journal and MDS I found here in the list. I hit a bug that stopped my MDS to start so I took the long way with reading the data. Everything went fine and I can even mount one of my CephFS now. That's a big relieve. But when I start scrub, I just get return code -116 and no scrub is initiatet. I didn't find that code in the docs. Can you help me? [ceph: root@ceph06 /]# ceph tell mds.mds01.ceph06.huavsw scrub start recursive 2023-04-29T10:46:36.926+0000 7ff676ff5700 0 client.79389355 ms_handle_reset on v2:192.168.23.66:6800/1133836262 2023-04-29T10:46:36.953+0000 7ff676ff5700 0 client.79389361 ms_handle_reset on v2:192.168.23.66:6800/1133836262 { "return_code": -116 } (I get the same error, no matter what kind of scrub I start) -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

2 2

import OSD after host OS reinstallation
by Tony Liu 29 Apr '23

29 Apr '23

Hi, The cluster is with Pacific and deployed by cephadm on container. The case is to import OSDs after host OS reinstallation. All OSDs are SSD who has DB/WAL and data together. Did some research, but not able to find a working solution. Wondering if anyone has experiences in this? What needs to be done before host OS reinstallation and what's after? Thanks! Tony

2 5

Block RGW request using Lua
by ondrej＠kuuk.la 28 Apr '23

28 Apr '23

Hello everyone, I've started playing with Lua scripting and would like to ask If anyone knows about a way to drop or close user request on the prerequest context. I would like to block creating buckets with dots in the name, but the use-case could be blocking certain operations, etc. I was able to come up with some like this if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = "InvalidBucketName" Request.Response.Message = "Dots are not allowed." end This works fine, but the bucket is created which is something that I don't want to do. As a dirty workaround, I've thought about changing the bucket name here to an already existing bucket, but the Request.Bucket.Name = "taken" doesn't seem to work as the log gives me an error "attempt to index a nil value (field 'Bucket')". Any help is much appreciated.

1 0

For suggestions and best practices on expanding Ceph cluster and removing old nodes
by huxiaoyu＠horebdata.cn 28 Apr '23

28 Apr '23

Dear Ceph folks, I would like to listen to your advice on the following topic: We have a 6-node Ceph cluster (for RGW usage only ) running on Luminous 12.2.12, and now will add 10 new nodes. Our plan is to phase out the old 6 nodes, and run RGW Ceph cluster with the new 10 nodes on Nautilus version。 I can think of two ways to achieve the above goal. The first method would be: 1) Upgrade the current 6-node cluster from Luminous 12.2.12 to Nautilus 14.2.22; 2) Expand the cluster with the 10 new nodes, and then re-balance; 3) After rebalance completes, remove the 6 old nodes from the cluster The second method would get rid of the procedure to upgrade the old 6-node from Luminous to Nautilus, because those 6 nodes will be phased out anyway, but then we have to deal with a hybrid cluster with 6-node on Luminous 12.2.12, and 10-node on Nautilus, and after re-balancing, we can remove the 6 old nodes from the cluster. Any suggestions, advice, or best practice would be highly appreciated. best regards, Samuel huxiaoyu(a)horebdata.cn

7 8

How to call cephfs-top
by E Taka 28 Apr '23

28 Apr '23

I'm using a dockerized Ceph 17.2.6 under Ubuntu 22.04. Presumably I'm missing a very basic thing, since this seems a very simple question: how can I call cephfs-top in my environment? It is not inckuded in the Docker Image which is accessed by "cephadm shell". And calling the version found in the source code always fails with "[errno 13] RADOS permission denied", even when using "--cluster" with the correct ID, "--conffile" and "--id". The auth user client.fstop exists, and "ceph fs perf stats" runs. What am I missing? Thanks!

2 1

Ceph stretch mode / POOL_BACKFILLFULL
by Kilian Ries 28 Apr '23

28 Apr '23

Hi, we run a ceph cluster in stretch mode with one pool. We know about this bug: https://tracker.ceph.com/issues/56650 https://github.com/ceph/ceph/pull/47189 Can anyone tell me what happens when a pool gets to 100% full? At the moment raw OSD usage is about 54% but ceph throws me a "POOL_BACKFILLFULL" error: $ ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 63 TiB 29 TiB 34 TiB 34 TiB 54.19 TOTAL 63 TiB 29 TiB 34 TiB 34 TiB 54.19 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 415 MiB 105 1.2 GiB 0.04 1.1 TiB vm_stretch_live 2 64 15 TiB 4.02M 34 TiB 95.53 406 GiB So the pool warning / calculation is just a bug, because it thinks its 50% of the total size. I know ceph will stop IO / set OSDs to read only if the hit a "backfillfull_ratio" ... but what will happen if the pool gets to 100% full ? Will IO still be possible? No limits / quotas are set on the pool ... Thanks Regards, Kilian

2 2

Ceph 16.2.12, particular OSD shows higher latency than others
by Zakhar Kirpichenko 28 Apr '23

28 Apr '23

Hi, I have a Ceph 16.2.12 cluster with uniform hardware, same drive make/model, etc. A particular OSD is showing higher latency than usual in `ceph osd perf`, usually mid to high tens of milliseconds while other OSDs show low single digits, although its drive's I/O stats don't look different from those of other drives. The workload is mainly random 4K reads and writes, the cluster is being used as Openstack VM storage. Is there a way to trace, which particular PG, pool and disk image or object cause this OSD's excessive latency? Is there a way to tell Ceph to I would appreciate any advice or pointers. Best regards, Zakhar

3 9

Massive OMAP remediation
by Ben.Zieglmeier 27 Apr '23

27 Apr '23

Hi All, We have a RGW cluster running Luminous (12.2.11) that has one object with an extremely large OMAP database in the index pool. Listomapkeys on the object returned 390 Million keys to start. Through bilog trim commands, we’ve whittled that down to about 360 Million. This is a bucket index for a regrettably unsharded bucket. There are only about 37K objects actually in the bucket, but through years of neglect, the bilog grown completely out of control. We’ve hit some major problems trying to deal with this particular OMAP object. We just crashed 4 OSDs when a bilog trim caused enough churn to knock one of the OSDs housing this PG out of the cluster temporarily. The OSD disks are 6.4TB NVMe, but are split into 4 partitions, each housing their own OSD daemon (collocated journal). We want to be rid of this large OMAP object, but are running out of options to deal with it. Reshard outright does not seem like a viable option, as we believe the deletion would deadlock OSDs can could cause impact. Continuing to run `bilog trim` 1000 records at a time has been what we’ve done, but this also seems to be creating impacts to performance/stability. We are seeking options to remove this problematic object without creating additional problems. It is quite likely this bucket is abandoned, so we could remove the data, but I fear even the deletion of such a large OMAP could bring OSDs down and cause potential for metadata loss (the other bucket indexes on that same PG). Any insight available would be highly appreciated. Thanks.

3 3

Bucket notification
by Szabo, Istvan (Agoda) 27 Apr '23

27 Apr '23

Hi, I'm trying to set a kafka endpoint for bucket object create operation notifications but the notification is not created in kafka endpoint. Settings seems to be fine because I can upload to the bucket objects when these settings are applied: NotificationConfiguration> <TopicConfiguration> <Id>bulknotif</Id> <Topic>arn:aws:sns:default::butcen</Topic> <Event>s3:ObjectCreated:*</Event> <Event>s3:ObjectRemoved:*</Event> </TopicConfiguration> </NotificationConfiguration> but it simply not created any message in kafka. This is my topic creation post request: https://xxx.local/? Action=CreateTopic& Name=butcen& kafka-ack-level=broker& use-ssl=true& push-endpoint=kafka://ceph:pw@xxx.local:9093 Am I missing something or it's definitely kafka issue? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Bucket empty after resharding on multisite environment
by Boris Behrens 27 Apr '23

27 Apr '23

Hi, I just resharded a bucket on an octopus multisite environment from 11 to 101. I did it on the master zone and it went through very fast. But now the index is empty. The files are still there when doing a radosgw-admin bucket radoslist --bucket-id Do I just need to wait or do I need to recover that somehow?

1 2

MDS recovery
by jack＠translucencebio.com 27 Apr '23

27 Apr '23

Hi All, We have a CephFS cluster running Octopus with three control nodes each running an MDS, Monitor, and Manager on Ubuntu 20.04. The OS drive on one of these nodes failed recently and we had to do a fresh install, but made the mistake of installing Ubuntu 22.04 where Octopus is not available. We tried to force apt to use the Ubuntu 20.04 repo when installing Ceph so that it would install Octopus, but for some reason Quincy was still installed. We re-integrated this node and it seemed to work fine for about a week until our cluster reported damage to an MDS daemon and placed our filesystem into a degraded state. cluster: id: 692905c0-f271-4cd8-9e43-1c32ef8abd13 health: HEALTH_ERR mons are allowing insecure global_id reclaim 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged noout flag(s) set 161 scrub errors Possible data damage: 24 pgs inconsistent 8 pgs not deep-scrubbed in time 4 pgs not scrubbed in time 6 daemons have recently crashed services: mon: 3 daemons, quorum database-0,file-server,webhost (age 12d) mgr: database-0(active, since 4w), standbys: webhost, file-server mds: cephfs:0/1 3 up:standby, 1 damaged osd: 91 osds: 90 up (since 32h), 90 in (since 5M) flags noout task status: data: pools: 7 pools, 633 pgs objects: 169.18M objects, 640 TiB usage: 883 TiB used, 251 TiB / 1.1 PiB avail pgs: 605 active+clean 23 active+clean+inconsistent 4 active+clean+scrubbing+deep 1 active+clean+scrubbing+deep+inconsistent We are not sure if the Quincy/Octopus version mismatch is the problem, but we are in the process of downgrading this node now to ensure all nodes are running Octopus. Before doing that, we ran the following commands to try and recover: $ cephfs-journal-tool --rank=cephfs:all journal export backup.bin $ sudo cephfs-journal-tool --rank=cephfs:all event recover_dentries summary: Events by type: OPEN: 29589 PURGED: 1 SESSION: 16 SESSIONS: 4 SUBTREEMAP: 127 UPDATE: 70438 Errors: 0 $ cephfs-journal-tool --rank=cephfs:0 journal reset: old journal was 170234219175~232148677 new journal start will be 170469097472 (2729620 bytes past old end) writing journal head writing EResetJournal entry done $ cephfs-table-tool all reset session All of our MDS daemons are down and fail to restart with the following errors: -3> 2023-04-20T10:25:15.072-0700 7f0465069700 -1 log_channel(cluster) log [ERR] : journal replay alloc 0x1000053af79 not in free [0x1000053af7d~0x1e8,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2,0x1000053b561~0x2,0x1000053b565~0x1de,0x1000053b938~0x1fd,0x1000053bd2a~0x4,0x1000053bf23~0x4,0x1000053c11c~0x4,0x1000053cd7b~0x158,0x1000053ced8~0xffffac3128] -2> 2023-04-20T10:25:15.072-0700 7f0465069700 -1 log_channel(cluster) log [ERR] : journal replay alloc [0x1000053af7a~0x1eb,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2], only [0x1000053af7d~0x1e8,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2] is in free [0x1000053af7d~0x1e8,0x1000053b35c~0x1f7,0x1000053b555~0x2,0x1000053b559~0x2,0x1000053b55d~0x2,0x1000053b561~0x2,0x1000053b565~0x1de,0x1000053b938~0x1fd,0x1000053bd2a~0x4,0x1000053bf23~0x4,0x1000053c11c~0x4,0x1000053cd7b~0x158,0x1000053ced8~0xffffac3128] -1> 2023-04-20T10:25:15.072-0700 7f0465069700 -1 /build/ceph-15.2.15/src/mds/journal.cc: In function 'void EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)' thread 7f0465069700 time 2023-04-20T10:25:15.076784-0700 /build/ceph-15.2.15/src/mds/journal.cc: 1513: FAILED ceph_assert(inotablev == mds->inotable->get_version()) ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x155) [0x7f04717a3be1] 2: (()+0x26ade9) [0x7f04717a3de9] 3: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x67e2) [0x560feaca36f2] 4: (EUpdate::replay(MDSRank*)+0x42) [0x560feaca5bd2] 5: (MDLog::_replay_thread()+0x90c) [0x560feac393ac] 6: (MDLog::ReplayThread::entry()+0x11) [0x560fea920821] 7: (()+0x8609) [0x7f0471318609] 8: (clone()+0x43) [0x7f0470ee9163] 0> 2023-04-20T10:25:15.076-0700 7f0465069700 -1 *** Caught signal (Aborted) ** in thread 7f0465069700 thread_name:md_log_replay ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable) 1: (()+0x143c0) [0x7f04713243c0] 2: (gsignal()+0xcb) [0x7f0470e0d03b] 3: (abort()+0x12b) [0x7f0470dec859] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f04717a3c3c] 5: (()+0x26ade9) [0x7f04717a3de9] 6: (EMetaBlob::replay(MDSRank*, LogSegment*, MDSlaveUpdate*)+0x67e2) [0x560feaca36f2] 7: (EUpdate::replay(MDSRank*)+0x42) [0x560feaca5bd2] 8: (MDLog::_replay_thread()+0x90c) [0x560feac393ac] 9: (MDLog::ReplayThread::entry()+0x11) [0x560fea920821] 10: (()+0x8609) [0x7f0471318609] 11: (clone()+0x43) [0x7f0470ee9163] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. At this point, we decided it's best to ask for some guidance before issuing any other recovery commands. Can anyone advise what we should do?

2 1

How to find the bucket name from Radosgw log?
by viplanghe6＠gmail.com 27 Apr '23

27 Apr '23

I find a log like this, and I thought the bucket name should be "photos": ....[2023-04-19 15:48:47.0.5541s] "GET /photos/shares/.... But I can not find it: radosgw-admin bucket stats --bucket photos failure: 2023-04-19 15:48:53.969 7f69dce49a80 0 could not get bucket info for bucket=photos (2002) Unknown error 2002 How does this happen? Thanks

3 2

Re: Disks are filling up
by Omar Siam 27 Apr '23

27 Apr '23

No this is a cephadm setup. Not rook. In the last days it is still deep scrubbing and filling up. We have to do something about it as it now impacts our K8s cluster (very slow cephfs access) and we are running out of (allocated) diskspace again. Some more details now that I had a few more days to think about our particular setup: * This is a setup with ESXi/vSphere virtualization. The ceph nodes are just some VMs. We don't have access to the bare servers or even direct access to the HDDs/SSDs ceph runs on. * The setup is "asymmetric": there are 2 nodes on SSDs and one on HDDs (they are all RAIDx with hardware controllers, but we have no say in this). I labeled all OSDs as HDDs (even when VMWare reported SSD). * We looked at the OSDs device usage and it is 100% (from the VMs point of view) for the HDDs (20% on everage for the SSD nodes). My suspision is: * deep scrubbing means every new write goes to unallocated space, no more overwrite/deleting while deep scrubbing. I didn't find it in the docs. Maybe I missed it, maybe that is common wisdom among the initiated. * We write more new data per second to cephfs than can be scrubbed so scrubbing never ends and the PGs fill up. We now ordered SSDs for the HDD only node to prevent this in the future. Meanwhile we need to do something so we think about moving the data in cephfs to a new PG that does not need deep scrubbing at the moment. Also we think about moving the OSD from the physical host that only has HDDs to one with SSDs ruining redundancy for a short while and hoping for the best Am 26.04.2023 um 02:28 schrieb A Asraoui: > > Omar, glad to see cephfs with kubernetes up and running.. did you guys > use rook to deploy this ?? > > Abdelillah > On Mon, Apr 24, 2023 at 6:56 AM Omar Siam <Omar.Siam(a)oeaw.ac.at> wrote: > > Hi list, > > we created a cluster for using cephfs with a kubernetes cluster. > Since a > few weeks now the cluster keeps filling up at an alarming rate > (100 GB per day). > This is while the most relevant pg is deep scrubbing and was > interupted > a few times. > > We use about 150G (du using the mounted filesystem) on the cephfs > filesystem and try not to use snapshots (.snap directories "exist" > but > are empty). > We do not understand why the pgs get bigger and bigger while cephfs > stays about the same size (overwrites on files certainly happen). > I suspect some snapshots mechanism. Any ideas how to debug this to > stop it? > > Maybe we should try to speed up the deep scrubbing somehow? > Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at |www.oeaw.ac.at/acdh

1 0

Veeam backups to radosgw seem to be very slow
by Boris Behrens 27 Apr '23

27 Apr '23

We have a customer that tries to use veeam with our rgw objectstorage and it seems to be blazingly slow. What also seems to be strange, that veeam sometimes show "bucket does not exist" or "permission denied". I've tested parallel and everything seems to work fine from the s3cmd/aws cli standpoint. Does anyone here ever experienced veeam problems with rgw? Cheers Boris

5 5

Rados gateway data-pool replacement.
by Gaël THEROND 26 Apr '23

26 Apr '23

Hi everyone, quick question regarding radosgw zone data-pool. I’m currently planning to migrate an old data-pool that was created with inappropriate failure-domain to a newly created pool with appropriate failure-domain. If I’m doing something like: radosgw-admin zone modify —rgw-zone default —data-pool <new_pool> Will data from the old pool be migrated to the new one or do I need to do something else to migrate those data out of the old pool? I’ve read a lot of mail archive with peoples willing to do that but I can’t get a clear answer from those archives. I’m running on nautilus release of it ever help. Thanks a lot! PS: This mail is a redo of the old one as I’m not sure the former one worked (missing tags).

3 4

Ceph 16.2.12, bluestore cache doesn't seem to be used much
by Zakhar Kirpichenko 26 Apr '23

26 Apr '23

Hi, I have a Ceph 16.2.12 cluster with hybrid OSDs (HDD block storage, DB/WAL on NVME). All OSD settings are default except, cache-related settings are as follows: osd.14 dev bluestore_cache_autotune true osd.14 dev bluestore_cache_size_hdd 4294967296 osd.14 dev bluestore_cache_size_ssd 4294967296 osd.14 advanced bluestore_default_buffered_write false osd.14 dev osd_memory_cache_min 2147483648 osd.14 basic osd_memory_target 17179869184 Other settings such as bluestore_cache_kv_ratio, bluestore_cache_meta_ratio, etc. are default. I.e. OSD memory target is set to 16 GB, bluestore cache is set to 4 GB for HDDs and SSDs, minimum cache size is 2 GB. When I dump memory pools of OSDs, bluestore cache doesn't seem to be actively used (https://pastebin.com/EpfFp85C) despite there's plenty of memory and the memory target is 16 GB, memory pools are around 2 GB and the total RSS of the OSD process is ~4.8 GB. There are 66 OSDs in the cluster and the situation is very similar with all of them. The OSDs are being used quite actively for both reads and writes, and I guess they could benefit from using more memory for caching, especially considering that we have lots of RAM available on each host. Is there a way to increase and/or tune OSD cache memory usage? I would appreciate any advice or pointers. Best regards, Zakhar

1 0

Object data missing, but metadata is OK (Quincy 17.2.3)
by Jeff Briden 26 Apr '23

26 Apr '23

We have a large cluster on Quincy 17.2.3 with a bucket holding 8.9 million small (15~20 MiB) objects. All the objects were multipart uploads from scripts using `aws s3 cp` The data is static (write-once, read-many) with no manual deletions and no new writes for months. We recently found 3 objects in this bucket that cannot be retrieved. The symptom is exactly the same as https://tracker.ceph.com/issues/47866 and https://bugzilla.redhat.com/show_bug.cgi?id=1892644 which were fixed a long time ago. Any form of listing (`aws s3 ls`, radosgw-admin object stat, radoslist, http head request, etc) returns good data, but the objects cannot be retrieved and rados -p ls shows the object data is missing. Any suggestions on how to troubleshoot this further?

1 0

How to control omap capacity？
by WeiGuo Ren 26 Apr '23

26 Apr '23

I have two osds. these osd are used to rgw index pool. After a lot of stress tests, these two osds were written to 99.90%. The full ratio (95%) did not take effect? I don't know much. Could it be that if the osd of omap is fully stored, it cannot be limited by the full ratio? ALSO I use ceph-bluestore-tool to expand it . Before I add a partition . But i failed, I dont know why. In my cluster every osd have 55GB (db val data in same device), ceph -v is 14.2.5. can anyone give me some idear to fix it?

2 1

Bug, pg_upmap_primaries.empty()
by Nguetchouang Ngongang Kevin 26 Apr '23

26 Apr '23

Good morning, i found a bug on ceph reef After installing ceph and deploying 9 osds with a cephfs layer. I got this error after many writing and reading operations on the ceph fs i deployed. ```{ "assert_condition": "pg_upmap_primaries.empty()", "assert_file": "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3593-g1e73409b/rpm/el8/BUILD/ceph-18.0.0-3593-g1e73409b/src/osd/OSDMap.cc", "assert_func": "void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const", "assert_line": 3239, "assert_msg": "/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3593-g1e73409b/rpm/el8/BUILD/ceph-18.0.0-3593-g1e73409b/src/osd/OSDMap.cc: In function 'void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const' thread 7f86cb8e5700 time 2023-04-26T12:25:12.278025+0000\n/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-3593-g1e73409b/rpm/el8/BUILD/ceph-18.0.0-3593-g1e73409b/src/osd/OSDMap.cc: 3239: FAILED ceph_assert(pg_upmap_primaries.empty())\n", "assert_thread_name": "msgr-worker-0", "backtrace": [ "/lib64/libpthread.so.0(+0x12cf0) [0x7f86d0d21cf0]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x55ce1794774b]", "/usr/bin/ceph-osd(+0x6368b7) [0x55ce179478b7]", "(OSDMap::encode(ceph::buffer::v15_2_0::list&, unsigned long) const+0x1229) [0x55ce183e0449]", "(MOSDMap::encode_payload(unsigned long)+0x396) [0x55ce17ae2576]", "(Message::encode(unsigned long, int, bool)+0x2e) [0x55ce1825dbee]", "(ProtocolV1::prepare_send_message(unsigned long, Message*, ceph::buffer::v15_2_0::list&)+0x54) [0x55ce184e5914]", "(ProtocolV1::write_event()+0x511) [0x55ce184f4ce1]", "(EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa64) [0x55ce182eb484]", "/usr/bin/ceph-osd(+0xfdf276) [0x55ce182f0276]", "/lib64/libstdc++.so.6(+0xc2b13) [0x7f86d0369b13]", "/lib64/libpthread.so.0(+0x81ca) [0x7f86d0d171ca]", "clone()" ], "ceph_version": "18.0.0-3593-g1e73409b", "crash_id": "2023-04-26T12:25:12.286947Z_55675d7c-7833-4e91-b0eb-6df705104c2e", "entity_name": "osd.0", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "0ffad2c4bc07caf68ff1e124d3911823bc6fa6f5772444754b7f0a998774c8fe", "timestamp": "2023-04-26T12:25:12.286947Z", "utsname_hostname": "node1-link-1", "utsname_machine": "x86_64", "utsname_release": "5.4.0-100-generic", "utsname_sysname": "Linux", "utsname_version": "#113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022" } ``` I really don't know what is this error for, Will appreciate any help. Cordially, -- Nguetchouang Ngongang Kevin ENS de Lyon https://perso.ens-lyon.fr/kevin.nguetchouang/

2 2

Ceph Leadership Team meeting minutes - 2023 April 26
by Casey Bodley 26 Apr '23

26 Apr '23

# ceph windows tests PR check will be made required once regressions are fixed windows build currently depends on gcc11 which limits use of c++20 features. investigating newer gcc or clang toolchain # 16.2.13 release final testing in progress # prometheus metric regressions https://tracker.ceph.com/issues/59505 related to previous discussion on 4/12 about quincy backports integration test coverage needed for ceph-exporter and the mgr module # lab update centos/rhel tests were failing due to problematic mirrorlists fixed in https://github.com/ceph/ceph-cm-ansible/pull/731 more sanity checks in progress at https://github.com/ceph/ceph-cm-ansible/pull/733 # cephalocon feedback dev summit etherpads: https://pad.ceph.com/p/cephalocon-dev-summit-2023 collect more notes here: https://pad.ceph.com/p/cephalocon-2023-brainstorm request for dev-focused longer term discussion could have specific user-focused and dev-focused sessions dense conference, hard to fit everything in 3 days could have longer component updates during conf, with time for questions perhaps 3 days of conf, dev-specific discussions a day before (no cfp, one big room, then option for breakout), user-feedback sessions during the normal con

1 0

Re: Move ceph to new addresses and hostnames
by Jan Marek 26 Apr '23

26 Apr '23

Hello all, today I moved ceph to HEALTH_OK state :-) 1) I had to restart MGR node, then my old c-osdx hostnames goes definitely away and all of OSDs from old machines are now orchestrated by 'ceph orch' command. 2) I've updated ceph* packages on the osd2 node to version 17.2.6, then I tried 'cephadm adopt' command once more and voila! It works like a charm. I will try to configure OSDs on the node 1 to adopt WAL and DB from prepared LVM... Maybe after upgrade to newer version of CEPH it will be OK? Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 1

OSD_TOO_MANY_REPAIRS on random OSDs causing clients to hang
by Thomas Hukkelberg 26 Apr '23

26 Apr '23

Hi all, Over the last 2 weeks we have experienced several OSD_TOO_MANY_REPAIRS errors that we struggle to handle in a non-intrusive manner. Restarting MDS + hypervisor that accessed the object in question seems to be the only way we can clear the error so we can repair the PG and recover access. Any pointers on how to handle this issue in a more gentle way than rebooting the hypervisor and failing the MDS would be welcome! The problem seems to only affect one specific pool (id 42) that is used for cephfs_data. This pool is our second cephfs data pool in this cluster. The data in the pool is accessible via LXC container via Samba and have the cephfs filesystem bind-mounted from hypervisor. Ceph is recently updated to version 16.2.11 (pacific) -- kernel version is 5.13.19-6-pve on OSD-hosts/samba-containers and 5.19.17-2-pve on MDS-hosts. The following warnings are issued: $ ceph health detail HEALTH_WARN 1 clients failing to respond to capability release; Too many repaired reads on 1 OSDs; Degraded data redundancy: 1/2648430 090 objects degraded (0.000%), 1 pg degraded; 1 slow ops, oldest one blocked for 608 sec, osd.34 has slow ops [WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release mds.hk-cephnode-65(mds.0): Client hk-cephnode-56 failing to respond to capability release client_id: 9534859837 [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs osd.34 had 9936 reads repaired [WRN] PG_DEGRADED: Degraded data redundancy: 1/2648430090 objects degraded (0.000%), 1 pg degraded pg 42.e2 is active+recovering+degraded+repair, acting [34,275,284] [WRN] SLOW_OPS: 1 slow ops, oldest one blocked for 608 sec, osd.34 has slow ops The logs for OSD.34 are flooded with these messages: root@hk-cephnode-53:~# tail /var/log/ceph/ceph-osd.34.log 2023-04-26T11:41:00.760+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 missing primary copy of 42:4703efac:::10003d86a99.00000001:head, will try copies on 275,284 2023-04-26T11:41:00.784+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.812+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.812+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 missing primary copy of 42:4703efac:::10003d86a99.00000001:head, will try copies on 275,284 2023-04-26T11:41:00.824+0200 7f03a821f700 -1 osd.34 1352563 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.9534859837.0:20412906 42.e2 42:4703efac:::10003d86a99.00000001:head [read 0~1048576 [307@0] out=1048576b] snapc 0=[] RETRY=5 ondisk+retry+read+known_if_redirected e1352553) 2023-04-26T11:41:00.824+0200 7f03a821f700 0 log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'qa-cephfs_data' : 1 ]) 2023-04-26T11:41:00.840+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.864+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head 2023-04-26T11:41:00.864+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 missing primary copy of 42:4703efac:::10003d86a99.00000001:head, will try copies on 275,284 2023-04-26T11:41:00.888+0200 7f03921f3700 -1 log_channel(cluster) log [ERR] : 42.e2 full-object read crc 0xebd673ed != expected 0xffffffff on 42:4703efac:::10003d86a99.00000001:head We have tried the following: - Restarting the OSD in question clears the error for a few seconds but then we also we get OSD_TOO_MANY_REPAIRS on OSDs with PGs that holds the object that have blocked I/O. - Trying to repair the PG seems to restart every 10 second and not actually do anything/progressing. (Is there a way to check repair progress?) - Restarting the MDS and hypervisor clears the error (the hypervisor hangs for several minutes before timing out). However if the object is requested again the error reoccurs. If we don't access the object we are able to eventually repair the PG. - Occasionally setting the primary-affinity to 0 for the primary OSD in the PG clears the error after restarting all affected OSD and we are able to repair the PG (unless the object is accessed during recovery) and access to the object is OK afterwards. - Finding and deleting the file pointing to the object (10003d86a99) and restarting OSDs will clear the error. - Killing the samba process that accessed the object does not clear the SLOW_OPS, and hence the error prevail - Normal scrubs have revealed a handfull of other PGs in the same pool (id 42) that are damaged and we are doing repairs without any problems. - We believe MDS_CLIENT_LATE_RELEASE and SLOW_OPS errors are symptoms of the fact that the I/O are blocked. - We have verified that there are no SMART errors of any kind on any of our disks in the cluster. - If we don't handle this issue rather promptly, we experience full lockup of the samba container and rebooting hypervisor seems to be the only cure. Trying to force unmount and remount cephfs does not help. This have now happened 6-7 times over the last 2 weeks and we suspect that a hardware or memory error on one of our nodes may have caused the objects to be written to disk with bad checksums. We have replaced the mainboard in one of our nodes that we might think is the culprit and are currently testing the memory. Can these random checksum errors be caused by anything else that we should investigate? It's a bit suspicious that the error only occurs on one specific pool? If the mainboard are to blame we should see these errors in more pools by now? Regardless we are stumped by how Ceph handles this error. Checksum-errors should not leave clients hanging like this? Should this be considered a bug? Is there a way to cancel the blocking I/O request to clear the error? And why is the PG flapping between active+recovering+degraded+repair, active+recovering+repair, active+clean+repair every few seconds? Any ideas on how to gracefully battle this problem? Thanks! --thomas Thomas Hukkelberg thomas(a)hovedkvarteret.no

3 3

Increase timeout for marking osd down
by Nicola Mori 26 Apr '23

26 Apr '23

Dear Ceph users, my cluster is made of very old machines on a Gbit ethernet. I see that sometimes some OSDs are marked down due to slow networking, especially on heavy network load like during recovery. This causes problems, for example PGs keeps being deactivated and activated as the OSDs are marked down and up (at least to my best understanding). So I'd need to know if there is some way to increase the timeout after which an OSD is marked down, to cope with my slow network. Thanks, Nicola

2 1

ceph pg stuck - missing on 1 osd how to proceed
by xadhoom76＠gmail.com 26 Apr '23

26 Apr '23

Hi to all Using ceph 17.2.5 i have 3 pgs in stuck state ceph pg map 8.2a6 osdmap e32862 pg 8.2a6 (8.2a6) -> up [88,100,59] acting [59,100] looking at it ho 88 ,100 and 59 i got that ceph pg ls-by-osd osd.100 | grep 8.2a6 8.2a6 211004 209089 0 0 174797925205 0 0 7075 active+undersized+degraded+remapped+backfilling 21m 32862'1540291 32862:3387785 [88,100,59]p88 [59,100]p59 2023-03-12T08:08:00.903727+0000 2023-03-12T08:08:00.903727+0000 6839 queued for deep scrub ceph pg ls-by-osd osd.59 | grep 8.2a6 8.2a6 211005 209084 0 0 174798941087 0 0 7076 active+undersized+degraded+remapped+backfilling 22m 32862'1540292 32862:3387798 [88,100,59]p88 [59,100]p59 2023-03-12T08:08:00.903727+0000 2023-03-12T08:08:00.903727+0000 6839 queued for deep scrub BUT ceph pg ls-by-osd osd.88 | grep 8.2a6 ---> NONE it is missing .... how to proceed ? Best regards

4 7

Dead node (watcher) won't timeout on RBD
by max＠netpunt.nl 26 Apr '23

26 Apr '23

Hey all, I recently had a k8s node failure in my homelab, and even though I powered it off (and it's done for, so it won't get back up), it still shows up as watcher in rbd status. ``` root@node0:~# rbd status kubernetes/csi-vol-3e7af8ae-ceb6-4c94-8435-2f8dc29b313b Watchers: watcher=10.0.0.103:0/1520114202 client.1697844 cookie=140289402510784 watcher=10.0.0.103:0/39967552 client.1805496 cookie=140549449430704 root@node0:~# ceph osd blocklist ls 10.0.0.103:0/0 2023-04-15T13:15:39.061379+0200 listed 1 entries ``` Even though the node is down & I have blocked it multiple times for hours, it won't disappear. Meaning, ceph-csi-rbd claims the image is mounted already (manually binding works fine, and can cleanly unbind as well, but can't unbind from a node that doesn't exist anymore). Is there any possibility to force kick an rbd client / watcher from ceph (e.g. switching the mgr / mon) or to see why this is not timing out? I found some historical mails & issues (related to rook, which I don't use) regarding a param `osd_client_watch_timeout` but can't find how that relates to the RBD images. Cheers, Max.

2 1

Ovirt integration with Ceph
by kushagra.gupta＠hsc.com 26 Apr '23

26 Apr '23

Hi Team, We are trying to integrate ceph with ovirt. We have deployed ovirt 4.4. We want to create a storage domain of POSIX compliant type for mounting a ceph based infrastructure in ovirt. We have done SRV based resolution in our DNS server for ceph mon nodes but we are unable to create a storage domain using that. We are able to manually mount the ceph-mon nodes using the following command on the deployment hosts: sudo mount -t ceph :/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 -o rw,name=foo,secret=AQABDzRkTaJCEhAAC7rC6E68ofwULnx6qX/VDA== [root@deployment-host mnt]# df -kh df: /run/user/0/gvfs: Transport endpoint is not connected Filesystem Size Used Avail Use% Mounted on [abcd:abcd:abcd::51]:6789,[abcd:abcd:abcd::52]:6789,[abcd:abcd:abcd::53]:6789:/volumes/xyz/conf/00593e1d-b674-4b00-a289-20bec06761c9 19G 0 19G 0% /rhev/data-center/mnt/:_volumes_xyz_conf_00593e1d-b674-4b00-a289-20bec06761c9 Query: 1. Could anyone help us out with storage domain creation in ovirt for SRV resolved ceph-mon nodes.

2 1

Could you please explain the PG concept
by wodel youchi 26 Apr '23

26 Apr '23

Hi, I am learning Ceph and I am having a hard time understanding PG and PG calculus . I know that a PG is a collection of objects, and that PG are replicated over the hosts to respect the replication size, but... In traditional storage, we use size in Gb, Tb and so on, we create a pool from a bunch of disks or raid arrays of some size then we create volumes of a certain size and use them. If the storage is full we add disks, then we extend our pools/volumes. The idea of size is simple to understand. Ceph, although it supports the notion of pool size in Gb, Tb ...etc. Pools are created using PGs, and now there is also the notion of % of data. When I use pg calc from ceph or from redhat, the generated yml file contains the % variable, but the commands file contains only the PGs, and when you are configuring 15% and 18% have the same number of PGs !!!!!!!!!!!!??? The pg calc encourages you to create a %data multiple of 100, in other words, it assumes that you know all your pools from the start. What if you won't consume all your raw disk space. What happens when you need to add a new pool? Also when you create several pools, and then execute ceph osd df tree, you can see that all pools show the raw size as a free space, it is like all pools share the same raw space regardless of their PG number. If someone can put some light on this concept and how to manage it wisely, because the documentation keeps saying that it's an important concept, that you have to pay attention when choosing the number of PGs for a pool from the start. Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free.www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

3 2

Reset a bucket in a zone
by Yixin Jin 25 Apr '23

25 Apr '23

Hi folks, Within a zonegroup, once a bucket is created, its metadata is sync-ed over to all zones. With bucket-level sync policy, however, its content may or may not be sync-ed over. To simplify the sync process, sometime I'd like to pick the bucket in a zone as the absolute truth and sync its content over to the bucket in another zone, which may have done some local deletions since the last time they were sync-ed. I don't want those local deletions to mess up with the planned sync. Is it possible to reset the bucket in this zone so it is in a "pristine" state and will receive everything from the source? Thanks,Yixin

1 0

Rados gateway lua script-package error lib64
by Thomas Bennett 25 Apr '23

25 Apr '23

Hi, I've noticed that when my lua script runs I get the following error on my radosgw container. It looks like the lib64 directory is not included in the path when looking for shared libraries. Copying the content of lib64 into the lib directory solves the issue on the running container. Here are more details: Apr 25 20:26:59 xxx-ceph-xxxx radosgw[60901]: req 2268223694354647302 0.000000000s Lua ERROR: /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*share*/lua/5.3/socket.lua:12: module 'socket.core' not found: no field package.preload['socket.core'] no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*share* /lua/5.3/socket/core.lua' no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*lib* /lua/5.3/socket/core.so' no file '/tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/*lib* /lua/5.3/socket.so' As mentioned the following command on the running radosgw container solves the issue for the running container: cp -ru /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib64/* /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib/ Cheers, Tom

1 0

cephadm grafana per host certificate
by Eugen Block 25 Apr '23

25 Apr '23

Hi *, I've set up grafana, prometheus and node-exporter on an adopted cluster (currently running 16.2.10) and was trying to enable ssl for grafana. As stated in the docs [1] there's a way to configure individual certs and keys per host: ceph config-key set mgr/cephadm/{hostname}/grafana_key -i $PWD/key.pem ceph config-key set mgr/cephadm/{hostname}/grafana_crt -i $PWD/certificate.pem So I did that, then ran 'ceph orch reconfig grafana' but I still get a bad cert error message: Apr 20 10:21:19 ceph01 conmon[3772491]: server.go:3160: http: TLS handshake error from <IP>:46084: remote error: tls: bad certificate It seems like the cephadm generated cert/key pair (mgr/cephadm/grafana_key; mgr/cephadm/grafana_crt) supersedes the per-host certs, and even after removing the generated cert/key (and then reconfigure) cephadm regenerates a them and leaves me with the same problem. Is this a known issue and what would be the fix? I didn't find anything on tracker, but I might have missed it. To confirm that my custom certs actually work I replaced the general cert with my custom cert and the error doesn't appear, I can see the grafana graphs in the dashboard. I could leave it like this, but if grafana would failover it wouldn't work anymore, of course. Any hints are greatly appreciated. Thanks, Eugen [1] https://docs.ceph.com/en/latest/cephadm/services/monitoring/#configuring-ss…

2 4

Move ceph to new addresses and hostnames
by Jan Marek 25 Apr '23

25 Apr '23

Hi there, I have ceph cluster created by ceph-volume - bluestore, in every node is 12 HDD and 1 NVMe, which is divided to 24 LVM partition for DB and WAL. I've turned this cluster to 'ceph orch' management, then I've moved to quincy release (now I'm using a 17.2.5 version). I had to move whole cluster to another addreses and another hostnames. MON, MGR and MDS goes without problem, but OSD was really pain process :-( Now I have cluster with this problem: # ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.cephfs.mon1.ulytsa mon1 running (11w) 3m ago 11w 5609M - 17.2.5 cc65afd6173a db1aa336263a mds.cephfs.mon2.zxhxqk mon2 running (11w) 3m ago 11w 33.1M - 17.2.5 cc65afd6173a 5b9ced4a4b71 mds.cephfs.mon3.rpkvlt mon3 running (11w) 3m ago 11w 32.4M - 17.2.5 cc65afd6173a 045e23f124aa mgr.mon1.buqyga mon1 *:8080 running (11w) 3m ago 11w 2300M - 17.2.5 cc65afd6173a 9577239527b5 mgr.mon2.goghws mon2 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 4fb1ae26765e mgr.mon3.slpgay mon3 *:8080 running (11w) 3m ago 11w 495M - 17.2.5 cc65afd6173a 06e491084a5e mon.mon1 mon1 running (11w) 3m ago 11w 1576M 2048M 17.2.5 cc65afd6173a 2f18c737faa9 mon.mon2 mon2 running (11w) 3m ago 11w 1598M 2048M 17.2.5 cc65afd6173a 31091cbbfb8e mon.mon3 mon3 running (11w) 3m ago 11w 1463M 2048M 17.2.5 cc65afd6173a 4d0b094c9ca1 osd.0 osd1 running (9w) 3m ago 10w 5133M 3745M 17.2.5 cc65afd6173a 3b28e48d3630 osd.1 osd1 running (7w) 3m ago 10w 5425M 3745M 17.2.5 cc65afd6173a 3336ccdfd232 osd.2 osd1 running (9w) 3m ago 10w 5223M 3745M 17.2.5 cc65afd6173a e8fc077aef59 osd.3 osd1 running (9w) 3m ago 10w 5050M 3745M 17.2.5 cc65afd6173a 4fbf34450237 osd.4 osd1 running (9w) 3m ago 10w 7526M 3745M 17.2.5 cc65afd6173a a4875c354540 osd.5 osd1 running (9w) 3m ago 10w 4854M 3745M 17.2.5 cc65afd6173a b006526228ae osd.6 osd1 running (9w) 3m ago 10w 6498M 3745M 17.2.5 cc65afd6173a 4c326271e188 osd.7 osd1 running (9w) 3m ago 10w 4410M 3745M 17.2.5 cc65afd6173a ca0f3ce31031 osd.8 osd1 running (9w) 3m ago 10w 7337M 3745M 17.2.5 cc65afd6173a 99269a832819 osd.9 osd1 running (9w) 3m ago 10w 4717M 3745M 17.2.5 cc65afd6173a f39ce0bb5316 osd.10 osd1 running (9w) 3m ago 10w 4295M 3745M 17.2.5 cc65afd6173a 0871793fa261 osd.11 osd1 running (9w) 3m ago 10w 5552M 3745M 17.2.5 cc65afd6173a 32a8b589b3bd osd.24 osd3 running (109m) 3m ago 6M 3306M 3745M 17.2.5 cc65afd6173a 466d80a55d96 osd.25 osd3 running (109m) 3m ago 6M 3145M 3745M 17.2.5 cc65afd6173a b1705621116a osd.26 osd3 running (109m) 3m ago 6M 3063M 3745M 17.2.5 cc65afd6173a c30253a1a83f osd.27 osd3 running (109m) 3m ago 6M 3257M 3745M 17.2.5 cc65afd6173a aa0a647d93f1 osd.28 osd3 running (109m) 3m ago 6M 2244M 3745M 17.2.5 cc65afd6173a d3c68ed6572b osd.29 osd3 running (109m) 3m ago 6M 3509M 3745M 17.2.5 cc65afd6173a 2c425b17abf7 osd.30 osd3 running (109m) 3m ago 6M 3814M 3745M 17.2.5 cc65afd6173a 44747256b34a osd.31 osd3 running (109m) 3m ago 6M 2958M 3745M 17.2.5 cc65afd6173a b7b7946fa24e osd.32 osd3 running (109m) 3m ago 6M 3016M 3745M 17.2.5 cc65afd6173a fc9c024fed4f osd.33 osd3 running (109m) 3m ago 6M 5366M 3745M 17.2.5 cc65afd6173a edc2dbd9c556 osd.34 osd3 running (109m) 3m ago 6M 4577M 3745M 17.2.5 cc65afd6173a 46d7668742cf osd.35 osd3 running (109m) 3m ago 6M 2538M 3745M 17.2.5 cc65afd6173a 96a15a9ad3d7 osd.36 osd4 running (103m) 3m ago 8w 2707M 3745M 17.2.5 cc65afd6173a adf884af609b osd.37 osd4 running (103m) 3m ago 6M 3347M 3745M 17.2.5 cc65afd6173a 8f824026c6ae osd.38 osd4 running (103m) 3m ago 6M 3377M 3745M 17.2.5 cc65afd6173a 2a70c0b860ff osd.39 osd4 running (103m) 3m ago 6M 2814M 3745M 17.2.5 cc65afd6173a 4d5833f1faaf osd.40 osd4 running (103m) 3m ago 6M 2633M 3745M 17.2.5 cc65afd6173a e2e492c4f4a6 osd.41 osd4 running (103m) 3m ago 6M 2141M 3745M 17.2.5 cc65afd6173a 80f67faf1238 osd.42 osd4 running (103m) 3m ago 6M 1527M 3745M 17.2.5 cc65afd6173a c1aef5891ad5 osd.43 osd4 running (103m) 3m ago 6M 1927M 3745M 17.2.5 cc65afd6173a b092b03f211d osd.44 osd4 running (103m) 3m ago 6M 3332M 3745M 17.2.5 cc65afd6173a 9309d93c80da osd.45 osd4 running (103m) 3m ago 6M 3172M 3745M 17.2.5 cc65afd6173a 8a47d7c155cd osd.46 osd4 running (103m) 3m ago 6M 2545M 3745M 17.2.5 cc65afd6173a 0550b66a837f osd.47 osd4 running (103m) 3m ago 6M 3010M 3745M 17.2.5 cc65afd6173a 52328564e503 osd.48 osd5 running (8w) 8m ago 8w 6054M 3745M 17.2.5 cc65afd6173a c5494c95566e osd.49 osd5 running (8w) 8m ago 8w 5648M 3745M 17.2.5 cc65afd6173a cebc3405950e osd.50 osd5 running (8w) 8m ago 8w 4898M 3745M 17.2.5 cc65afd6173a 44bd2f018476 osd.51 osd5 running (8w) 8m ago 8w 5066M 3745M 17.2.5 cc65afd6173a 008cb9d89e68 osd.52 osd5 running (8w) 8m ago 8w 6325M 3745M 17.2.5 cc65afd6173a db3d01bbfce7 osd.53 osd5 running (8w) 8m ago 8w 6040M 3745M 17.2.5 cc65afd6173a 215c70cf7ca3 osd.54 osd5 running (8w) 8m ago 8w 5147M 3745M 17.2.5 cc65afd6173a 92a4bc3eae20 osd.55 osd5 running (8w) 8m ago 8w 6590M 3745M 17.2.5 cc65afd6173a ce07fe507b47 osd.56 osd5 running (8w) 8m ago 8w 3838M 3745M 17.2.5 cc65afd6173a 540cf969419d osd.57 osd5 running (8w) 8m ago 8w 4820M 3745M 17.2.5 cc65afd6173a 89653e57e85c osd.58 osd5 running (8w) 8m ago 8w 5016M 3745M 17.2.5 cc65afd6173a c19d06b09533 osd.59 osd5 running (8w) 8m ago 8w 6547M 3745M 17.2.5 cc65afd6173a 8c387d3d06a6 osd.60 osd6 running (99m) 3m ago 6M 2050M 3745M 17.2.5 cc65afd6173a 90c1d1745480 osd.61 osd6 running (99m) 3m ago 6M 2694M 3745M 17.2.5 cc65afd6173a 32da4b4cec45 osd.62 osd6 running (99m) 3m ago 6M 2608M 3745M 17.2.5 cc65afd6173a 43788b932105 osd.63 osd6 running (99m) 3m ago 6M 4048M 3745M 17.2.5 cc65afd6173a 4a6863748bc5 osd.64 osd6 running (99m) 3m ago 6M 2542M 3745M 17.2.5 cc65afd6173a c49dd095aa64 osd.65 osd6 running (99m) 3m ago 6M 2796M 3745M 17.2.5 cc65afd6173a cdb2bed58cf3 osd.66 osd6 running (99m) 3m ago 6M 2129M 3745M 17.2.5 cc65afd6173a c17db58e047f osd.67 osd6 running (99m) 3m ago 6M 3559M 3745M 17.2.5 cc65afd6173a 46f2527c6112 osd.68 osd6 running (99m) 3m ago 6M 2552M 3745M 17.2.5 cc65afd6173a c4491c1b8de9 osd.69 osd6 running (99m) 3m ago 6M 2750M 3745M 17.2.5 cc65afd6173a 9e03c1d5351e osd.70 osd6 running (99m) 3m ago 6M 1849M 3745M 17.2.5 cc65afd6173a 3454e09aab70 osd.72 osd6 running (99m) 3m ago 8w 5368M 3745M 17.2.5 cc65afd6173a 0c1ca9a8677d # ceph orch host ls HOST ADDR LABELS STATUS iscsi1 192.168.6.166 iscsi2 192.168.6.167 mon1 192.168.7.208 mon2 192.168.7.209 mon3 192.168.7.210 osd1 192.168.7.214 osd2 192.168.7.215 osd3 192.168.7.216 osd4 192.168.7.217 osd5 192.168.7.218 osd6 192.168.7.219 11 hosts in cluster # ceph health detail HEALTH_WARN 12 stray daemon(s) not managed by cephadm; 3 stray host(s) with 35 daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 12 stray daemon(s) not managed by cephadm stray daemon osd.12 on host osd2 not managed by cephadm stray daemon osd.13 on host osd2 not managed by cephadm stray daemon osd.14 on host osd2 not managed by cephadm stray daemon osd.15 on host osd2 not managed by cephadm stray daemon osd.16 on host osd2 not managed by cephadm stray daemon osd.17 on host osd2 not managed by cephadm stray daemon osd.18 on host osd2 not managed by cephadm stray daemon osd.19 on host osd2 not managed by cephadm stray daemon osd.20 on host osd2 not managed by cephadm stray daemon osd.21 on host osd2 not managed by cephadm stray daemon osd.22 on host osd2 not managed by cephadm stray daemon osd.23 on host osd2 not managed by cephadm [WRN] CEPHADM_STRAY_HOST: 3 stray host(s) with 35 daemon(s) not managed by cephadm stray host c-osd3 has 12 stray daemons: ['osd.24', 'osd.25', 'osd.26', 'osd.27', 'osd.28', 'osd.29', 'osd.30', 'osd.31', 'osd.32', 'osd.33', 'osd.34', 'osd.35'] stray host c-osd4 has 12 stray daemons: ['osd.36', 'osd.37', 'osd.38', 'osd.39', 'osd.40', 'osd.41', 'osd.42', 'osd.43', 'osd.44', 'osd.45', 'osd.46', 'osd.47'] stray host c-osd6 has 11 stray daemons: ['osd.60', 'osd.61', 'osd.62', 'osd.63', 'osd.64', 'osd.65', 'osd.66', 'osd.67', 'osd.68', 'osd.69', 'osd.70'] Comments: 1) c-osd3, c-osd4, c-osd6 are old names (with old addresses) which are not in cluster now, as you can see from 'ceph orch ps' listing. 2) On the osd1 machine I've tried create osd daemons bych 'ceph orch', but I cannot create with one NVMe for DB and WAL together, thus I have OSD without cached DB and WAL. 3) On the osd2, there is a OSD daemons created by ceph-volume 4) On the other hosts I've renamed host, moved to another IP address, 'ceph orch rm c-osdx', 'ceph orch add osdx', in LVM changed all c-osdx tags to osdx tags, now I've changed in /var/lib/ceph/uuid/osd.x/unit.run and unit.poststop every c-osdx to osdx. And now questions: Is there a way to definitelly remove c-osd3, c-osd4 and c-osd6 from 'ceph health detail' and manage this host via 'ceph orch'? On the osd2, there is a problem to adopt OSD created by ceph-volume to ceph orchestrator: cephadm adopt --style legacy -n osd.12 Pulling container image quay.io/ceph/ceph:v17... Found online OSD at //var/lib/ceph/osd/ceph-12/fsid objectstore_type is bluestore Stopping old systemd unit ceph-osd@12... Disabling old systemd unit ceph-osd@12... Moving data... Traceback (most recent call last): File "/usr/sbin/cephadm", line 9468, in <module> main() File "/usr/sbin/cephadm", line 9456, in main r = ctx.func(ctx) File "/usr/sbin/cephadm", line 2135, in _default_image return func(ctx) File "/usr/sbin/cephadm", line 6582, in command_adopt command_adopt_ceph(ctx, daemon_type, daemon_id, fsid) File "/usr/sbin/cephadm", line 6755, in command_adopt_ceph os.rmdir(data_dir_src) OSError: [Errno 39] Directory not empty: '//var/lib/ceph/osd/ceph-12' Thanks a lot. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

1 0

Re: Bucket sync policy
by Yixin Jin 25 Apr '23

25 Apr '23

Actually, "bucket sync run" somehow made it worse since now the destination zone shows "bucket is caught up with source" from "bucket sync status" even though it clearly missed an object. On Monday, April 24, 2023 at 02:37:46 p.m. EDT, Yixin Jin <yjin77(a)yahoo.ca> wrote: An update: After creating and enabling the bucket sync policy, I ran "bucket sync markers" and saw that each shard had the status of "init". The run "bucket sync run" in the end marked the status to be "incremental-sync", which seems to go through full-sync stage. However, the lone object in the source zone wasn't synced over to the destination zone. I actually used gdb to walk through radosgw-admin to run "bucket sync run". It seems not to do anything for full-sync and it printed a log saying "finished iterating over all available prefixes:...", which actually broke off the do-while loop after the call to prefix_handler.revalidate_marker(&list_marker). This call returned false because it couldn't find rules from the sync pipe. I haven't drilled deeper to see why it didn't get rules, whatever it means. Nevertheless, the workaround with "bucket sync run" doesn't seem to work, at least not with Quincy. Regards,Yixin On Monday, April 24, 2023 at 12:37:24 p.m. EDT, Soumya Koduri <skoduri(a)redhat.com> wrote: On 4/24/23 21:52, Yixin Jin wrote: > Hello ceph gurus, > > We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. > > > > Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync? yes this is known_issue yet to be addressed with bucket level sync policy ( - https://tracker.ceph.com/issues/57489 ). The interim workaround to sync existing objects is to either * create new objects (or) * execute "bucket sync run" after creating/enabling the bucket policy. Please note that this issue is specific to only bucket policy but doesn't exist for sync-policy set at zonegroup level. Thanks, Soumya _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 3

Bucket sync policy
by Yixin Jin 25 Apr '23

25 Apr '23

Hello ceph gurus, We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync? I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. I also tried "bucket sync init" and "bucket sync run" via radosgw-admin. They don't seem to do what I expected. They simply mark z1 as not behind anymore but the single object still lives in z0 only. I wonder how mature this sync policy feature is for production use. Thanks,Yixin

3 2

pacific 16.2.13 point release
by Yuri Weinstein 24 Apr '23

24 Apr '23

We want to do the next urgent point release for pacific 16.2.13 ASAP. The tip of the current pacific branch will be used as a base for this release and we will build it later today. Dev leads - if you have any outstanding PRs that must be included pls merged them now. Thx YuriW

3 2

Re: Bucket sync policy
by Yixin Jin 24 Apr '23

24 Apr '23

Re-post my rely to make sure it goes through the mailing list. ----- Forwarded Message ----- From: Yixin Jin <yjin77(a)yahoo.ca>To: Soumya Koduri <skoduri(a)redhat.com>Sent: Monday, April 24, 2023 at 02:37:46 p.m. EDTSubject: Re: [ceph-users] Re: Bucket sync policy An update: After creating and enabling the bucket sync policy, I ran "bucket sync markers" and saw that each shard had the status of "init". The run "bucket sync run" in the end marked the status to be "incremental-sync", which seems to go through full-sync stage. However, the lone object in the source zone wasn't synced over to the destination zone. I actually used gdb to walk through radosgw-admin to run "bucket sync run". It seems not to do anything for full-sync and it printed a log saying "finished iterating over all available prefixes:...", which actually broke off the do-while loop after the call to prefix_handler.revalidate_marker(&list_marker). This call returned false because it couldn't find rules from the sync pipe. I haven't drilled deeper to see why it didn't get rules, whatever it means. Nevertheless, the workaround with "bucket sync run" doesn't seem to work, at least not with Quincy. Regards,Yixin On Monday, April 24, 2023 at 12:37:24 p.m. EDT, Soumya Koduri <skoduri(a)redhat.com> wrote: On 4/24/23 21:52, Yixin Jin wrote: > Hello ceph gurus, > > We are trying bucket-specific sync policy feature with Quincy release and we encounter something strange. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-specific sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. > > > > Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. Why wasn’t there a full sync and how can one force a full sync? yes this is known_issue yet to be addressed with bucket level sync policy ( - https://tracker.ceph.com/issues/57489 ). The interim workaround to sync existing objects is to either * create new objects (or) * execute "bucket sync run" after creating/enabling the bucket policy. Please note that this issue is specific to only bucket policy but doesn't exist for sync-policy set at zonegroup level. Thanks, Soumya _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

v16.2.12 Pacific (hot-fix) released
by Yuri Weinstein 24 Apr '23

24 Apr '23

We're happy to announce the 12th hot-fix release in the Pacific series. https://ceph.io/en/news/blog/2023/v16-2-12-pacific-released/ Notable Changes --------------- This is a hotfix release that resolves several performance flaws in ceph-volume, particularly during osd activation (https://tracker.ceph.com/issues/57627) Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.12.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 5a2d516ce4b134bfafc80c4274532ac0d56fc1e2

4 4

Bucket sync policy
by yjin77＠yahoo.ca 24 Apr '23

24 Apr '23

Hello, ceph gurus, We are trying multisite sync policy feature with Quincy release and we encounter something strange, which we cannot solve even after combing through the internet for clues. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm, a single zonegroup and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-level sync policy being “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. So, why wasn’t there a full sync and how can one force a full sync? I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. Thanks, Yixin

1 0

Disks are filling up
by Omar Siam 24 Apr '23

24 Apr '23

Hi list, we created a cluster for using cephfs with a kubernetes cluster. Since a few weeks now the cluster keeps filling up at an alarming rate (100 GB per day). This is while the most relevant pg is deep scrubbing and was interupted a few times. We use about 150G (du using the mounted filesystem) on the cephfs filesystem and try not to use snapshots (.snap directories "exist" but are empty). We do not understand why the pgs get bigger and bigger while cephfs stays about the same size (overwrites on files certainly happen). I suspect some snapshots mechanism. Any ideas how to debug this to stop it? Maybe we should try to speed up the deep scrubbing somehow? ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) cluster: id: ece0290c-cd32-11ec-a0e2-005056a9dd02 health: HEALTH_WARN 1 MDSs report slow metadata IOs 3 nearfull osd(s) 13 pool(s) nearfull services: mon: 3 daemons, quorum acdh-gluster-hdd3,acdh-gluster-hdd1,acdh-gluster-hdd2 (age 3d) mgr: acdh-gluster-hdd3.kzsplh(active, since 5d), standbys: acdh-gluster-hdd2.kiotbg, acdh-gluster-hdd1.ywgyfx mds: 1/1 daemons up, 2 standby osd: 3 osds: 3 up (since 4d), 3 in (since 7w) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 292 pgs objects: 167.25M objects, 1.2 TiB usage: 3.3 TiB used, 1.2 TiB / 4.5 TiB avail pgs: 290 active+clean 1 active+clean+scrubbing+deep 1 active+clean+scrubbing io: client: 58 MiB/s rd, 3.6 MiB/s wr, 51 op/s rd, 148 op/s wr rancher-ceph-fs - 227 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ceph-mds.acdh-gluster-hdd1.pqydya Reqs: 68 /s 793k 792k 102k 210k POOL TYPE USED AVAIL rancherFsPoolMetadata metadata 160G 329G rancherFsPoolDefaultData data 2268k 329G rancherFsPoolMainData data 2584G 658G STANDBY MDS ceph-mds.acdh-gluster-hdd2.zfleqe ceph-mds.acdh-gluster-hdd3.etaobl MDS version: ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable) (rancherFsPoolMainData is a 2+1 erasure encoded pool) --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 TOTAL 4.5 TiB 1.2 TiB 3.3 TiB 3.3 TiB 73.46 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 0 B 0 0 B 0 331 GiB rancher-rbd-erasure 2 32 8.4 GiB 2.16k 13 GiB 1.25 661 GiB rancher-rbd-meta 3 32 55 B 11 36 KiB 0 331 GiB rancherFsPoolMetadata 4 32 53 GiB 5.18M 160 GiB 13.88 331 GiB rancherFsPoolDefaultData 5 1 29 KiB 80.00M 2.2 MiB 0 331 GiB rancherFsPoolMainData 6 1 1.7 TiB 82.08M 2.5 TiB 72.23 661 GiB .rgw.root 7 32 1.3 KiB 4 48 KiB 0 331 GiB default.rgw.log 8 32 3.6 KiB 209 408 KiB 0 331 GiB default.rgw.control 9 32 0 B 8 0 B 0 331 GiB default.rgw.meta 10 32 3.8 KiB 11 124 KiB 0 331 GiB default.rgw.buckets.index 11 32 2.4 MiB 33 7.2 MiB 0 331 GiB default.rgw.buckets.non-ec 12 32 0 B 0 0 B 0 331 GiB default.rgw.buckets.data 14 1 55 GiB 16.57k 83 GiB 7.70 661 GiB HEALTH_WARN 1 MDSs report slow metadata IOs; 3 nearfull osd(s); 13 pool(s) nearfull [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.ceph-mds.acdh-gluster-hdd1.pqydya(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 306 secs [WRN] OSD_NEARFULL: 3 nearfull osd(s) osd.0 is near full osd.2 is near full osd.3 is near full [WRN] POOL_NEARFULL: 13 pool(s) nearfull pool 'device_health_metrics' is nearfull pool 'rancher-rbd-erasure' is nearfull pool 'rancher-rbd-meta' is nearfull pool 'rancherFsPoolMetadata' is nearfull pool 'rancherFsPoolDefaultData' is nearfull pool 'rancherFsPoolMainData' is nearfull pool '.rgw.root' is nearfull pool 'default.rgw.log' is nearfull pool 'default.rgw.control' is nearfull pool 'default.rgw.meta' is nearfull pool 'default.rgw.buckets.index' is nearfull pool 'default.rgw.buckets.non-ec' is nearfull pool 'default.rgw.buckets.data' is nearfull (near full is set to 0.66) -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

1 0

Troubleshooting cephadm OSDs aborting start
by André Gemünd 24 Apr '23

24 Apr '23

Dear Ceph-users, we have trouble with a Ceph cluster after a full shutdown. A couple of OSDs don't start anymore, exiting with SIGABRT very quickly. With debug logs and lots of work (I find cephadm clusters hard to debug btw) we received the following stack trace: debug -16> 2023-04-14T11:52:17.617+0000 7f10ab4d2700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/osd/PGLog.h: In function 'void PGLog::IndexedLog::add(const pg_log_entry_t&, bool)' thread 7f10ab4d2700 time 2023-04-14T11:52:17.614095+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7/rpm/el8/BUILD/ceph-16.2.7/src/osd/PGLog.h: 607: FAILED ceph_assert(head.version == 0 || e.version.version > head.version) ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x55b2dafc7b7e] 2: /usr/bin/ceph-osd(+0x56ad98) [0x55b2dafc7d98] 3: (bool PGLog::append_log_entries_update_missing<pg_missing_set<true> >(hobject_t const&, std::__cxx11::list<pg_log_entry_t, mempool::pool_allocator<(mempool::pool_index_t)22, pg_log_entry_t> > const&, bool, PGLog::IndexedLog*, pg_missing_set<true>&, PGLog::LogEntryHandler*, DoutPrefixProvider const*)+0xc19) [0x55b2db1bb6b9] 4: (PGLog::merge_log(pg_info_t&, pg_log_t&&, pg_shard_t, pg_info_t&, PGLog::LogEntryHandler*, bool&, bool&)+0xee2) [0x55b2db1adf22] 5: (PeeringState::merge_log(ceph::os::Transaction&, pg_info_t&, pg_log_t&&, pg_shard_t)+0x75) [0x55b2db33c165] 6: (PeeringState::Stray::react(MLogRec const&)+0xcc) [0x55b2db37adec] 7: (boost::statechart::simple_state<PeeringState::Stray, PeeringState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd5) [0x55b2db3a6e65] 8: (boost::statechart::state_machine<PeeringState::PeeringMachine, PeeringState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x5b) [0x55b2db18ef6b] 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PeeringCtx&)+0x2d1) [0x55b2db1839e1] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x29c) [0x55b2db0fde5c] 11: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x55b2db32d0e6] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x55b2db0efd48] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x55b2db7615b4] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b2db764254] 15: /lib64/libpthread.so.0(+0x817f) [0x7f10cef1117f] 16: clone() debug -15> 2023-04-14T11:52:17.618+0000 7f10b64e8700 3 osd.70 72507 handle_osd_map epochs [72507,72507], i have 72507, src has [68212,72507] debug -14> 2023-04-14T11:52:17.619+0000 7f10b64e8700 3 osd.70 72507 handle_osd_map epochs [72507,72507], i have 72507, src has [68212,72507] debug -13> 2023-04-14T11:52:17.619+0000 7f10ac4d4700 5 osd.70 pg_epoch: 72507 pg[18.7( v 64162'106 (0'0,64162'106] local-lis/les=72506/72507 n=14 ec=17104/17104 lis/c=72506/72480 les/c/f=72507/72481/0 sis=72506 pruub=9.160680771s) [70,86,41] r=0 lpr=72506 pi=[72480,72506)/1 crt=64162'106 lcod 0'0 mlcod 0'0 active+wait pruub 12.822580338s@ mbc={}] exit Started/Primary/Active/Activating 0.011269 7 0.000114 # ceph versions { "mon": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "osd": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 92 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 103 } } Another thing is that things like `ceph -s`, `ceph osd tree`, `rbd ls`, etc. work, but `ceph orch ps` (or generally any orch commands) simply hang forever, seemingly in a futex waiting on a socket to the mons. If anyone has any ideas how we could get those OSDs back online, I'd be very grateful for any hints. I'm also on slack. Greetings -- André Gemünd, Leiter IT / Head of IT Fraunhofer-Institute for Algorithms and Scientific Computing andre.gemuend(a)scai.fraunhofer.de Tel: +49 2241 14-4199 /C=DE/O=Fraunhofer/OU=SCAI/OU=People/CN=Andre Gemuend

2 2

How to replace an HDD in a OSD with shared SSD for DB/WAL
by Tao LIU 23 Apr '23

23 Apr '23

HI, I build a Ceph Cluster with cephadm. Every cehp node has 4 OSDs. These 4 OSD were build with 4 HDD (block) and 1 SDD (DB). At present , one HDD is broken, and I am trying to replace the HDD,and build the OSD with the new HDD and the free space of the SDD. I did the follows: #ceph osd stop osd.23 #ceph osd out osd.23 #ceph osd crush remove osd.23 #ceph osd rm osd.23 #ceph orch daemon rm osd.23 --force #lvremove /dev/ceph-ae21e618-601e-4273-9185-99180edb8453/osd-block-96eda371-1a3f-4139-9123-24ec1ba362c4 #wipefs -af /dev/sda #lvremove /dev/ceph-e50203a6-8b8e-480f-965c-790e21515395/osd-db-70f7a032-cf2c-4964-b979-2b90f43f2216 #ceph orch daemon add osd compute11:data_devices=/dev/sda,db_devices=/dev/sdc,osds_per_device=1 The OSD can be built, but is always down. Is there anyting that I missed during the building? Thank you very much! Regards, LIUTao

3 2

radosgw-admin bucket rm fails
by James Turner 22 Apr '23

22 Apr '23

I've got buckets that are visible to radosgw-admin bucket list but not to an S3 client and the delete fails.. e.g.: radosgw-admin bucket rm --bucket=appsshared1 2023-04-22T19:43:55.705-0400 7f9a87616680 0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-2 2023-04-22T19:43:55.705-0400 7f9a87616680 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -2 for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) Similarly, trying to delete the owning user fails: radosgw-admin user rm --uid=apps_shared_user --purge-data could not remove user: unable to remove user, unable to delete user data 2023-04-22T19:54:34.266-0400 7f84c598c680 0 ERROR: cls_bucket_list_unordered error in rgw_rados_operate (bucket list op), r=-2 2023-04-22T19:54:34.266-0400 7f84c598c680 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -2 for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) As does trying to fix the bucket: radosgw-admin bucket check --bucket appsshared1 --fix 2023-04-22T19:56:17.510-0400 7f981eddd680 0 int RGWRados::cls_bucket_list_ordered(const DoutPrefixProvider*, RGWBucketInfo&, int, const rgw_obj_index_key&, const string&, const string&, uint32_t, bool, uint16_t, RGWRados::ent_map_t&, bool*, bool*, rgw_obj_index_key*, optional_yield, RGWBucketListNameFilter): CLSRGWIssueBucketList for :appsshared1[07c644e6-08ba-4c2a-aa43-46c47fcaa3d2.2815988.1]) failed The release is quincy - Any ideas on how to approach the problem? -- Jim Turner

1 0

advise on adding RGW and NFS/iSCSI on proxmox
by MartijnF 22 Apr '23

22 Apr '23

Hello, i'm maintaining a 3 small infrastructures in NL. 7 primary, 4 secondary and 3 testing proxmox nodes (hyperconverged). Till now we only use the embedded RBD facilities of proxmox using ceph 17.2.5 To facilitate S3/swift and persistent storage to our internal k8s cluster i am looking around for best-practices like: - should i install it on-top of proxmox-hosts or create a few ha-VM's with public and private lan-segment to avoid upgrade errors. - can i use the new ceph-dashboard to create and install mentioned services needed? Should i virtualize a proxmox or other OS for this? I can imagine it is not their primary focus. A lot of information can be found when using google-foo but it is hard to find answers reflecting my setup. Thanks in advance for any pointers! Martijn

1 0

[sync policy] multisite bucket full sync
by yjin77＠yahoo.ca 21 Apr '23

21 Apr '23

Hi folks, We are trying multisite sync policy feature with Quincy release and we encounter something strange. Maybe our understanding of sync policy is incorrect. I hope the community could help us uncover the mystery. Our test setup is very simple. I use mstart.sh to spin up 3 clusters, configure them with a single realm "world", a single zonegroup "zg" and 3 zones – z0, z1, z2, with z0 being the master. I created a zonegroup-level sync policy with status “allowed”, a symmetrical flow among all 3 zones and a pipe allowing all zones to all zones. I created a single bucket “test-bucket” at z0 and uploaded a single object to it. By now, there should be no sync since the policy status is “allowed” only and I can see the single file only exist in z0 and “bucket sync status” shows the sync is actually disabled. Finally, I created a bucket-level sync policy with status “enabled” and a pipe between z0 and z1 only. I expected that sync should be kicked off between z0 and z1 and I did see from “sync info” that there are sources/dests being z0/z1. “bucket sync status” also shows the source zone and source bucket. At z0, it shows everything is caught up but at z1 it shows one shard is behind for data sync, which is expected since that only object exists in z0 but not in z1. Now, here comes the strange part. Although z1 shows there is one shard behind, it doesn’t seem to make any progress on syncing it. It doesn’t seem to do any full sync at all since “bucket sync status” shows “full sync: 0/11 shards”. There hasn’t been any full sync since otherwise, z1 should have that only object. It is stuck in this condition forever until I make another upload on the same object. I suspect the update of the object triggers a new data log, which triggers the sync. My questions are: 1. With bucket-specific sync policy, why wasn’t there a full sync and how can one force a full sync? 2. Does the sync policy only work datalogs, without which no action it may take? I wonder if it would explain why it didn't sync even though it knows it is behind? BTW, I also tried “sync error list” and they are all empty. I also tried to apply the fix in https://tracker.ceph.com/issues/57853, although I am not sure if it is relevant. The fix didn’t change the behavior that we observed. Thanks in advance, Yixin

1 0

Re: pg_autoscaler using uncompressed bytes as pool current total_bytes triggering false POOL_TARGET_SIZE_BYTES_OVERCOMMITTED warnings?
by Christian Rohmann 21 Apr '23

21 Apr '23

Hey ceph-users, may I ask (nag) again about this issue? I am wondering if anybody can confirm my observations? I raised a bug https://tracker.ceph.com/issues/54136, but apart from the assignment to a dev a while ago here was not response yet. Maybe I am just holding it wrong, please someone enlighten me. Thank you and with kind regards Christian On 02/02/2022 20:10, Christian Rohmann wrote: > > Hey ceph-users, > > > I am debugging a mgr pg_autoscaler WARN which states a > target_size_bytes on a pool would overcommit the available storage. > There is only one pool with value for target_size_bytes (=5T) defined > and that apparently would consume more than the available storage: > > --- cut --- > # ceph health detail > HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes > [WRN] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED: 1 subtrees have > overcommitted pool target_size_bytes > Pools ['backups', 'images', 'device_health_metrics', '.rgw.root', > 'redacted.rgw.control', 'redacted.rgw.meta', 'redacted.rgw.log', > 'redacted.rgw.otp', 'redacted.rgw.buckets.index', > 'redacted.rgw.buckets.data', 'redacted.rgw.buckets.non-ec'] overcommit > available storage by 1.011x due to target_size_bytes 15.0T on pools > ['redacted.rgw.buckets.data']. > --- cut --- > > > But then looking at the actual usage it seems strange that 15T (5T * 3 > replicas) should not fit onto the remaining 122 TiB AVAIL: > > > --- cut --- > # ceph df detail > --- RAW STORAGE --- > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 293 TiB 122 TiB 171 TiB 171 TiB 58.44 > TOTAL 293 TiB 122 TiB 171 TiB 171 TiB 58.44 > > --- POOLS --- > POOL ID PGS STORED (DATA) (OMAP) > OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS > QUOTA BYTES DIRTY USED COMPR UNDER COMPR > backups 1 1024 92 TiB 92 TiB 3.8 MiB > 28.11M 156 TiB 156 TiB 11 MiB 64.77 28 TiB N/A > N/A N/A 39 TiB 123 TiB > images 2 64 1.7 TiB 1.7 TiB 249 KiB > 471.72k 5.2 TiB 5.2 TiB 748 KiB 5.81 28 TiB N/A > N/A N/A 0 B 0 B > device_health_metrics 19 1 82 MiB 0 B 82 > MiB 43 245 MiB 0 B 245 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > .rgw.root 21 32 23 KiB 23 KiB 0 B > 25 4.1 MiB 4.1 MiB 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.control 22 32 0 B 0 B 0 B > 8 0 B 0 B 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.meta 23 32 1.7 MiB 394 KiB 1.3 > MiB 1.38k 237 MiB 233 MiB 3.9 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.log 24 32 53 MiB 500 KiB 53 > MiB 7.60k 204 MiB 47 MiB 158 MiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.otp 25 32 5.2 KiB 0 B 5.2 > KiB 0 16 KiB 0 B 16 KiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.buckets.index 26 32 1.2 GiB 0 B 1.2 > GiB 7.46k 3.5 GiB 0 B 3.5 GiB 0 28 TiB > N/A N/A N/A 0 B 0 B > redacted.rgw.buckets.data 27 128 3.1 TiB 3.1 TiB 0 B > 3.53M 9.5 TiB 9.5 TiB 0 B 10.11 28 TiB N/A > N/A N/A 0 B 0 B > redacted.rgw.buckets.non-ec 28 32 0 B 0 B 0 B > 0 0 B 0 B 0 B 0 28 TiB N/A > N/A N/A 0 B 0 B > --- cut --- > > > I then looked at how those values are determined at > https://github.com/ceph/ceph/blob/9f723519257eca039126a20aa6a2a7d2dbfb5dba/…. > Apparently "total_bytes" are compared with the capacity of the > root_map. I added a debug line and found that the total in my cluster > was already at: > > total=325511007759696 > > so in excess of 300 TiB - Looking at "ceph df" again this usage seems > strange. > > > > Looking at how this total is calculated at > https://github.com/ceph/ceph/blob/9f723519257eca039126a20aa6a2a7d2dbfb5dba/…, > you see that the larger value (max) of "actual_raw_used" vs. > "target_bytes*raw_used_rate" is determined and then summed up. > > > I dumped the values for all pools my cluster with yet another line of > debug code: > > ---cut --- > pool_id 1 - actual_raw_used=303160109187420.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 2 - actual_raw_used=5714098884702.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 19 - actual_raw_used=256550760.0, target_bytes=0 raw_used_rate=3.0 > pool_id 21 - actual_raw_used=71433.0, target_bytes=0 raw_used_rate=3.0 > pool_id 22 - actual_raw_used=0.0, target_bytes=0 raw_used_rate=3.0 > pool_id 23 - actual_raw_used=5262798.0, target_bytes=0 raw_used_rate=3.0 > pool_id 24 - actual_raw_used=162299940.0, target_bytes=0 raw_used_rate=3.0 > pool_id 25 - actual_raw_used=16083.0, target_bytes=0 raw_used_rate=3.0 > pool_id 26 - actual_raw_used=3728679936.0, target_bytes=0 > raw_used_rate=3.0 > pool_id 27 - actual_raw_used=10035209699328.0, > target_bytes=5497558138880 raw_used_rate=3.0 > pool_id 28 - actual_raw_used=0.0, target_bytes=0 raw_used_rate=3.0 > --- cut --- > > > All values but those of pool_id 1 (backups) make sense. For backups > it's just reporting a MUCH larger actual_raw_used value than what is > shown via ceph df. > The only difference of that pool compared to the others is the enabled > compression: > > > --- cut --- > # ceph osd pool get backups compression_mode > compression_mode: aggressive > --- cut --- > > > Apparently there already was a similar issue > (https://tracker.ceph.com/issues/41567) with a resulting commit > (https://github.com/ceph/ceph/commit/dd6e752826bc762095be4d276e3c1b8d31293eb0) > > changing which from "bytes_used" to the "stored" field for > "pool_logical_used". > > But how does that take compressed (away) data into account? Does > "bytes_used" count all the "stored" bytes, summing up all uncompressed > bytes for pools with compression? > This surely must be a bug then, as those bytes are not really > "actual_raw_used". > > > > I was about to raise a bug, but I wanted to ask here on the ML first > if I misunderstood the mechanisms at play here. > Thanks and with kind regards, > > > Christian >

1 0

17.2.6 dashboard: unable to get RGW dashboard working
by Michel Jouvin 21 Apr '23

21 Apr '23

Hi, I just upgraded in 17.2.6 but in fact I had the same problem in 16.2.10. I'm trying to configure the Ceph dashboard to monitor the RGWs (object gateways used as S3 gw). Our cluster has 2 RGW realms (eros, fink) with 1 zonegroup per realm (p2io-eros and p2io-fink respectively) and 1 zone per zonegroup (eros-z1 and fink-z1 respectively). RGW have been configured without SSL due to the bug encountered in 16.2.10 (not sure if it still applies, use of IP addresses that don't match the certificate). There is a dashboard user in each realm and its access/secret keys are matching what is in mgr/dashboard/RGW_API_ACCESS_KEY and mgr/dashboard/RGW_API_SECRET_KEY (a dict with one entry per realm). Every attempt to access the RGW daemons section of the dashboard results in : ----- The Object Gateway Service is not configured Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"InvalidAccessKeyId","RequestId":"tx00000b4c2212597168bf0-00644116e3' b'-edf7c3-eros-z1","HostId":"edf7c3-eros-z1-ijclab"}') ---- In the mgr logs, I have not been able to find more than this error. In the RGW logs, for every dashboard attempt, I find: ------ Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: ====== starting new request req=0x7f2dcdb0b710 ===== Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: req 13025009517641239536 0.000000000s op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: ====== req done req=0x7f2dcdb0b710 op status=0 http_status=403 latency=0.000000000s ====== Apr 20 12:41:39 valvd-rgw1.lal.in2p3.fr radosgw[1969455]: beast: 0x7f2dcdb0b710: 134.158.194.10 - - [20/Apr/2023:10:41:39.843 +0000] "GET /admin/metadata/user?myself HTTP/1.1" 403 126 - "python-requests/2.20.0" - latency=0.000000000s ------ In the logs above, I find a little bit strange the "HostID" value ending with "eros-z1-ijclab". I'd expected it to be the zone name "eros-z1". The zone has been renamed a couple of times and it may be an old name, despite the zone information in radosgw-admin shows nothing with this name (in particular the API name has been renamed to match the zone name). Anway the RGW works perfectly and the dashboard request seems to reach the RGW so I don't have the feeling it is important... Any suggestion on how to troubleshoot what happens would be appreciated. Thanks in advance. Best regards, Michel

1 1

deploying Ceph using FQDN for MON / MDS Services
by Lokendra Rathour 21 Apr '23

21 Apr '23

Hi Team, We have a ceph cluster with 3 storage nodes: 1. storagenode1 - abcd:abcd:abcd::21 2. storagenode2 - abcd:abcd:abcd::22 3. storagenode3 - abcd:abcd:abcd::23 The requirement is to mount ceph using the domain name of MON node: Note: we resolved the domain name via DNS server. For this we are using the command: ``` mount -t ceph [storagenode.storage.com]:6789:/ /backup -o name=admin,secret=AQCM+8hjqzuZEhAAcuQc+onNKReq7MV+ykFirg== ``` We are getting the following logs in /var/log/messages: ``` Jan 24 17:23:17 localhost kernel: libceph: resolve 'storagenode.storage.com' (ret=-3): failed Jan 24 17:23:17 localhost kernel: libceph: parse_ips bad ip ' storagenode.storage.com:6789' ``` We also tried mounting ceph storage using IP of MON which is working fine. Query: Could you please help us out with how we can mount ceph using FQDN. My /etc/ceph/ceph.conf is as follows: [global] ms bind ipv6 = true ms bind ipv4 = false mon initial members = storagenode1,storagenode2,storagenode3 osd pool default crush rule = -1 fsid = 7969b8a3-1df7-4eae-8ccf-2e5794de87fe mon host = [v2:[abcd:abcd:abcd::21]:3300,v1:[abcd:abcd:abcd::21]:6789],[v2:[abcd:abcd:abcd::22]:3300,v1:[abcd:abcd:abcd::22]:6789],[v2:[abcd:abcd:abcd::23]:3300,v1:[abcd:abcd:abcd::23]:6789] public network = abcd:abcd:abcd::/64 cluster network = eff0:eff0:eff0::/64 [osd] osd memory target = 4294967296 [client.rgw.storagenode1.rgw0] host = storagenode1 keyring = /var/lib/ceph/radosgw/ceph-rgw.storagenode1.rgw0/keyring log file = /var/log/ceph/ceph-rgw-storagenode1.rgw0.log rgw frontends = beast endpoint=[abcd:abcd:abcd::21]:8080 rgw thread pool size = 512 -- ~ Lokendra skype: lokendrarathour

7 20

How to replace an HDD in a OSD with shared SSD for DB/WAL
by enochlew＠gmail.com 21 Apr '23

21 Apr '23

HI, I build a Ceph Cluster with cephadm. Every cehp node has 4 OSDs. These 4 OSD were build with 4 HDD (block) and 1 SDD (DB). At present , one HDD is broken, and I am trying to replace the HDD,and build the OSD with the new HDD and the free space of the SDD. I did the follows: #ceph osd stop osd.23 #ceph osd out osd.23 #ceph osd crush remove osd.23 #ceph osd rm osd.23 #ceph orch daemon rm osd.23 --force #lvremove /dev/ceph-ae21e618-601e-4273-9185-99180edb8453/osd-block-96eda371-1a3f-4139-9123-24ec1ba362c4 #wipefs -af /dev/sda #lvremove /dev/ceph-e50203a6-8b8e-480f-965c-790e21515395/osd-db-70f7a032-cf2c-4964-b979-2b90f43f2216 #ceph orch daemon add osd compute11:data_devices=/dev/sda,db_devices=/dev/sdc,osds_per_device=1 The OSD can be built, but is always down. Is there anyting that I missed during the building? Thank you very much! Regards, LIUTao

1 0

I am unable to execute 'rbd map xxx' as it returns the error 'rbd: map failed: (5) Input/output error'.
by siriusa51＠outlook.com 21 Apr '23

21 Apr '23

$ rbd map xxx rbd: sysfs write failed 2023-04-21 11:29:13.786418 7fca1bfff700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2023-04-21 11:29:13.786456 7fca1b7fe700 -1 librbd::ImageState: 0x55a60108a040 failed to open image: (5) Input/output error rbd: error opening image xxx: (5) Input/output error In some cases useful info is found in syslog - try "dmesg | tail" or so. rbd: map failed: (5) Input/output error the command 'dmesg | tail' did not show any useful results. and other command: $ rbd info xxx 2023-04-21 11:33:10.223701 7f3547fff700 -1 librbd::image::OpenRequest: failed to retrieve image id: (5) Input/output error 2023-04-21 11:33:10.223739 7f35477fe700 -1 librbd::ImageState: 0x5647d5cfeeb0 failed to open image: (5) Input/output error rbd: error opening image xxx: (5) Input/output error i know header id is c2c061579478fe ,so i get omapvals: $ rados -p rbd listomapvals rbd_header.c2c061579478fe features value (8 bytes) : 00000000 01 00 00 00 00 00 00 00 |........| 00000008 object_prefix value (27 bytes) : 00000000 17 00 00 00 72 62 64 5f 64 61 74 61 2e 63 32 63 |....rbd_data.c2c| 00000010 30 36 31 35 37 39 34 37 38 66 65 |061579478fe| 0000001b order value (1 bytes) : 00000000 16 |.| 00000001 size value (8 bytes) : 00000000 00 00 00 00 71 02 00 00 |....q...| 00000008 snap_seq value (8 bytes) : 00000000 00 00 00 00 00 00 00 00 |........| 00000008 I scanned all existing data blocks using the command 'rados -p rbd list | grep c2c061579478fe', and found many 'rbd_data' blocks. I suspect that the loss of important blocks caused this situation. Currently, my plan is to obtain all 'rbd' blocks, concatenate them into an XFS file, and then perform a repair. However, as the data size is huge, I am still experimenting with this method. Are there any other feasible methods to repair this data? rbd_data: rbd_data.c2c061579478fe.0000000000000000 rbd_data.c2c061579478fe.0000000000000001 rbd_data.c2c061579478fe.0000000000000002 rbd_data.c2c061579478fe.0000000000000003 rbd_data.c2c061579478fe.0000000000000004 rbd_data.c2c061579478fe.0000000000000005 rbd_data.c2c061579478fe.0000000000000006 rbd_data.c2c061579478fe.0000000000000007 rbd_data.c2c061579478fe.0000000000000008 rbd_data.c2c061579478fe.0000000000000009 rbd_data.c2c061579478fe.000000000000000a rbd_data.c2c061579478fe.000000000000000b rbd_data.c2c061579478fe.000000000000000c rbd_data.c2c061579478fe.000000000000000d rbd_data.c2c061579478fe.000000000000000e rbd_data.c2c061579478fe.000000000000000f rbd_data.c2c061579478fe.0000000000000010 rbd_data.c2c061579478fe.0000000000000011 ... rbd_data.c2c061579478fe.000000000009c3f6 rbd_data.c2c061579478fe.000000000009c3f7 rbd_data.c2c061579478fe.000000000009c3fa rbd_data.c2c061579478fe.000000000009c3fb rbd_data.c2c061579478fe.000000000009c3fc rbd_data.c2c061579478fe.000000000009c3fd rbd_data.c2c061579478fe.000000000009c3fe rbd_data.c2c061579478fe.000000000009c3ff rbd_header.c2c061579478fe thanks

1 0

Can I delete rgw log entries?
by Richard Bade 20 Apr '23

20 Apr '23

Hi Everyone, I've been having trouble finding an answer to this question. Basically I'm wanting to know if stuff in the .log pool is actively used for anything or if it's just logs that can be deleted. In particular I was wondering about sync logs. In my particular situation I have had some tests of zone sync setup, but now I've removed the secondary zone and pools. My primary zone is filled with thousands of logs like this: data_log.71 data.full-sync.index.e2cf2c3e-7870-4fc4-8ab9-d78a17263b4f.47 meta.full-sync.index.7 datalog.sync-status.shard.e2cf2c3e-7870-4fc4-8ab9-d78a17263b4f.13 bucket.sync-status.f3113d30-ecd3-4873-8537-aa006e54b884:{bucketname}:default.623958784.455 I assume that because I'm not doing any sync anymore I can delete all the sync related logs? Is anyone able to confirm this? What about if the sync is running? Are these being written and read from and therefore must be left alone? It seems like these are more of a status than just a log and that deleting them might confuse the sync process. If so, does that mean that the log pool is not just output that can be removed as needed? Are there perhaps other things in there that need to stay? Regards, Richard

3 3

Be careful with primary-temp to balance primaries ...
by Stefan Kooman 20 Apr '23

20 Apr '23

Hi, A word of caution for Ceph operators out there. Be careful with "ceph osd primary-temp" command. TL;DR: with primary_temp active, a CRUSH change might CRASH your OSDs ... and they won't come back online after a restart (in almost all cases). The bug is described in this tracker [1], and fixed with this PR [2] (thanks Igor!). The longer story is that we were inspired by the work done on the read-balancer [3] and wondered if we could leverage this on older clusters. It turned out this is indeed possible by using the "primary-temp" command, instead of "pg-primary-temp" that will be available from Reef onward. We compiled a main version of the "osdmaptool", fed it OSD maps from pacific clusters and have it calculate the optimal primary PG distributions. Then we replaced the pg-primary-temp command with "primary-temp" and applied the commands. That worked as expected. However, we hit a bug [1] in the Ceph code that could not handle a situation when there were changes in the CRUSH map when primary_temp where active. We added a new storage node to the cluster that triggered this condition as soon as we put it in the proper failure domain. It tried to make an OSD primary that was not in the active set anymore, and hence crashed (with a Segmentation Fault most often, or aborted). This resulted in multiple (many) OSD crashes across the failure domains and basically took down the whole cluster. If the Reef / main read-balancer code can suffer from the same bug is as of yet unknown (at least to us). We will try to build a Reef test cluster and find out. For those of you who want to know how we handled this incident can read the RFO ([4] in dutch, [5] in English). Gr. Stefan [1]: https://tracker.ceph.com/issues/59491?next_issue_id=59490 [2]: https://github.com/ceph/ceph/pull/51160 [3]: https://github.com/ljflores/ceph_read_balancer_2023 [4]: https://www.bit.nl/uploads/images/PDF-Files/RFO-20230314-185335.pdf [5]: https://www.bit.nl/uploads/images/PDF-Files/2023.04.20%20RFO_Ceph Cluster_185335_EN.pdf

2 1

User + Dev Monthly Meetup cancelled
by Laura Flores 20 Apr '23

20 Apr '23

Hi users, The User + Dev monthly meetup is cancelled today since some folks are still at or returning from Cephalocon. See you next month! - Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

quincy user metadata constantly changing versions on multisite slave with radosgw roles
by Christopher Durham 20 Apr '23

20 Apr '23

Hi, I am using 17.2.6 on rocky linux for both the master and the slave site I noticed that: radosgw-admin sync status often shows that the metadata sync is behind a minute or two on the slave. This didn't make sense, as the metadata isn't changing as far as I know. radosgw-admin mdlog list (on slave) showed me that there were user changes in metadata very often. After doing a little research, here is a scenario I was able to develop: 1. user continually writes to a bucket he owns, pointing his aws cli (for this test) to the master side endpoint as specified in ~/.aws/config while this is running, do the following on the slave: radosgw-admin metadata get user:<username> This always shows the same result for the user, no changes. The data gets to the slave side bucket. 2. Restart the continual copy, but this time use a role that the user is a member of via profile in ~/.aws/credentials and .~/aws/config, again writing to the master endpoint as specified in ~/.aws/config aws --profile <profile> s3 cp <file> s3://<bucket>/file where profile is set up to use a role definiiton. The data gets to the bucket on both sides. I do not have access if I do not use the role profille (to confirm I set it up right) However, while doing this second test, if I continually do: radosgw-admin metadata get user:<username> on the slave, I see a definite increase in versions. Here is a section of the json output: "key": "user:<username>", "ver": { "tag": "somestring", "ver:" 12145} the 12145 value increases over and over again, and the mtime value in the json output increases too based on the current date. (not shown here). The same value, when queried on the master side, remains 1, and the mtime value is the date the user was created or last changed by an admin. If I write a file only once, the vers value increases by 1 too, but not sure if the increase in vers is ncessarily 1:1 with the number of writes. This seems to be the source of my continual metadata lag. Am I missing something? I suspect that this has been happening for awhile and not specific to 17.2.6 as I just upgraded and the ver value is over 12000 for the user that I discovered. (I used a python script to sync roles between master and slave prior to 17.2.6. Now roles are replicated in 17.2.6). -Chris

2 1

Some hint for a DELL PowerEdge T440/PERC H750 Controller...
by Marco Gaiarin 20 Apr '23

20 Apr '23

We are testing an experimental Ceph cluster with server and controller at subject. The controller have not an HBA mode, but only a 'NonRAID' mode, come sort of 'auto RAID0' configuration. We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well, and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad (particulary, very low IOPS). There's some hint for disk/controller configuration/optimization? Thanks. -- Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso... mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)

7 18

[ceph 17.2.6] unable to create rbd snapshots for images with erasure code data-pool
by Reto Gysi 20 Apr '23

20 Apr '23

Dear Ceph Users, After upgrading from version 17.2.5 to 17.2.6 I no longer seem to be able to create snapshots of images that have an erasure coded datapool. root@zephir:~# rbd snap create ceph-dev@backup_20230417 Creating snap: 10% complete...failed. rbd: failed to create snapshot: (95) Operation not supported root@zephir:~# rbd info ceph-dev rbd image 'ceph-dev': size 10 GiB in 2560 objects order 22 (4 MiB objects) snapshot_count: 11 id: d2f3d287f13c7b data_pool: ecpool_hdd block_name_prefix: rbd_data.7.d2f3d287f13c7b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Wed Nov 23 17:01:03 2022 access_timestamp: Sun Apr 16 17:20:58 2023 modify_timestamp: Wed Nov 23 17:01:03 2022 root@zephir:~# Before the upgrade I was able to create snapshots of this pool: SNAPID NAME SIZE PROTECTED TIMESTAMP 1538 ceph-dev_2023-03-05T02:00:09.030+01:00 10 GiB Sun Mar 5 02:00:14 2023 1545 ceph-dev_2023-03-06T02:00:03.832+01:00 10 GiB Mon Mar 6 02:00:05 2023 1903 ceph-dev_2023-04-05T03:22:01.315+02:00 10 GiB Wed Apr 5 03:22:02 2023 1909 ceph-dev_2023-04-05T03:35:56.748+02:00 10 GiB Wed Apr 5 03:35:57 2023 1915 ceph-dev_2023-04-05T03:37:23.778+02:00 10 GiB Wed Apr 5 03:37:24 2023 1930 ceph-dev_2023-04-06T02:00:06.159+02:00 10 GiB Thu Apr 6 02:00:07 2023 1940 ceph-dev_2023-04-07T02:00:05.913+02:00 10 GiB Fri Apr 7 02:00:06 2023 1952 ceph-dev_2023-04-08T02:00:06.534+02:00 10 GiB Sat Apr 8 02:00:07 2023 1964 ceph-dev_2023-04-09T02:00:06.430+02:00 10 GiB Sun Apr 9 02:00:07 2023 2003 ceph-dev_2023-04-11T02:00:09.750+02:00 10 GiB Tue Apr 11 02:00:10 2023 2014 ceph-dev_2023-04-12T02:00:09.528+02:00 10 GiB Wed Apr 12 02:00:10 2023 root@zephir:~# I have looked through the release notes of 17.2.6 but couldn't find anything obvious regarding rbd and ec pools. Does anyone else have this problem? Do I need to change some config setting, or was this feature disabled or is it a bug? Ceph version info: root@zephir:~# ceph orch upgrade check --ceph_version 17.2.6 { "needs_update": {}, "non_ceph_image_daemons": [ "promtail.debian", "node-exporter.debian", "promtail.zephir", "grafana.zephir", "node-exporter.zephir", "prometheus.zephir", "loki.zephir", "alertmanager.zephir" ], "target_digest": " quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "target_id": "9cea3956c04b2d889b91b58f957577fcb4eacd3852df073e3e2567f159fcdbf8", "target_name": "quay.io/ceph/ceph:v17.2.6", "target_version": "ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)", "up_to_date": [ "iscsi.rbd.debian.ijztzu", "mds.jellyfin.debian.dcsocv", "mon.debian", "osd.13", "osd.6", "mds.backups.debian.runngh", "mds.rgysi.debian.uhgqen", "crash.debian", "mgr.debian.sookxi", "iscsi.rbd.zephir.viqahd", "osd.1", "mds.jellyfin.zephir.iqywsn", "osd.12", "osd.7", "osd.2", "crash.zephir", "rgw.default.zephir.jqmick", "mds.backups.zephir.ygigch", "osd.0", "osd.4", "mon.zephir", "mgr.zephir.enywvy", "mds.rgysi.zephir.diylss", "osd.3", "osd.10", "osd.5", "osd.8", "osd.11" ] } root@zephir:~# root@zephir:~# rbd --version ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) root@zephir:~# Cheers Reto Gysi

5 21

Ceph rgw ldap user acl and quotas
by Ignazio Cassano 19 Apr '23

19 Apr '23

Hello Everyone, We are going to install ceph object storage with ldap autenticazione. We would like to know if acl and quotas on objects and buckets work fine with ldap users. Thanks Ignazio

1 0

Unprivileged Ceph containers
by Stephen Smith6 19 Apr '23

19 Apr '23

Hey folks – just thought I’d ask here – today cephadm creates unit files for Ceph containers with --privileged in the “podman run” call. I saw https://github.com/ceph/ceph-container/blob/main/src/daemon/README.md but I’m not sure how this relates to Ceph when deployed with cephadm? Thanks, Eric

1 0

upgrading from el7 / nautilus
by Marc 19 Apr '23

19 Apr '23

Sorry for addressing this again. But I think there are quite a few still with Nautilus, that are planning such upgrade. Nautilus is currently available for el7, el8 Octopus is currently available for el7, el8 Pacific is currently available for el8, el9 Quincy is currently available for el8, el9 el7 eol 2024 I would like to end up at el9 with a the Pacific release that does not have these performance issues[1] However with the currently offered ceph releases I am more or less forced to an upgrade path of: upgrade hosts from el7 -> el8, upgrade from nautilus -> pacific, upgrade el8 -> el9 - disadvantage upgrading hosts with some co-located services to el8 and then again to el9 - disadvantage having to wait with el9 upgrade until pacific performance issue is fixed. the alternative I have been looking at is building Nautilus for el9 (centos9 stream) which seems to go ok (site-test-packages is not building though). This would result in a more efficient upgrade of: upgrade hosts from el7 -> el9, upgrade from nautilus -> pacific - advantage is only 1 os upgrade. - advantage is I can stay on Nautilus until the pacifc performance issue is fixed. Up until now I can't remember having to much issues with building things from source. However with this storage I would like to have more reassurance. 1. can anyone share experiences with building for different os? 2. what is with all these build warnings, is this common? 3. is it possible that @ceph they can build these rpms and have this go through their default (automated) testing/deploy procedures? I assume with such nice testing facilities this should not take that much time, not? 4. Can anyone recommend some testing procedures? I have a 3 node test cluster. I was thinking of upgrading one host to el9 and run these el9 Nautilus rpms on it, see how it holds. ps. I am not really interested yet in adding complexity with podman. pps. Would be nice if someone from ceph could advice on this. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg19012.html

2 1

HBA or RAID-0 + BBU
by Murilo Morais 19 Apr '23

19 Apr '23

Good evening everyone! Guys, about the P420 RAID controller, I have a question about the operation mode: What would be better: HBA or RAID-0 with BBU (active write cache)? Thanks in advance!

5 4

pacific el7 rpms
by Marc 19 Apr '23

19 Apr '23

Is there ever going to be rpms in https://download.ceph.com/rpm-pacific/el7/

2 2

Consequence of maintaining hundreds of clones of a single RBD image snapshot
by Eyal Barlev 19 Apr '23

19 Apr '23

Hello, My use-case involves creating hundreds of clones (~1,000) of a single RBD image snapshot. I assume watchers exist for each clone, due to the copy-on-write nature of clones. Should I expect a penalty for maintaining such a large number of clones: cpu, memory, performance? If such penalty does exist, we might opt to flatten some of the clones. Is consistency guaranteed during the flattening process? In other words, can I write to a clone while it is being flattened? Perspectivus

3 2

metadata sync
by Christopher Durham 17 Apr '23

17 Apr '23

Hi, After doing a 'radosgw-admin metadata sync init' on a secondary site in a 2-cluster multisite configuration, (See my previous post on doing a quincy upgade with sts roles manually synced between primary and secondary), and letting it sync all metadata from a master zone, I get the following as a large omap object warning on the secondary site: (from ceph health detail): "1 large objects found in pool <zone>.rgw.log" And in ceph.log: "Large omap object found. Object: 87.adcf1188:::meta.full-sync.index.0:head PG 87.abcdfdff (87.0) Key count: 1234567 Size (bytes) 0" I know about large omap objects, and because this is in the log pool, I should be able to delete it with the rados utility. (And do a deep scrub on the pg. Am I correct? Looks like old bucket metadata is still there even though the buckets have been deleted. -Chris

1 0

MGR Memory Leak in Restful
by Chris Palmer 17 Apr '23

17 Apr '23

We've hit a memory leak in the Manager Restful interface, in versions 17.2.5 & 17.2.6. On our main production cluster the active MGR grew to about 60G until the oom_reaper killed it, causing a successful failover and restart of the failed one. We can then see that the problem is recurring, actually on all 3 of our clusters. We've traced this to when we enabled full Ceph monitoring by Zabbix last week. The leak is about 20GB per day, and seems to be proportional to the number of PGs. For some time we just had the default settings, and no memory leak, but had not got around to finding why many of the Zabbix items were showing as Access Denied. We traced this to the MGR's MON CAPS which were "mon 'profile mgr'". The MON logs showed recurring: log_channel(audit) log [DBG] : from='mgr.284576436 192.168.xxx.xxx:0/2356365' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: access denied Changing the MGR CAPS to "mon 'allow *'" and restarting the MGR immediately allowed that to work, and all the follow-on REST calls worked. log_channel(audit) log [DBG] : from='mgr.283590200 192.168.xxx.xxx:0/1779' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: dispatch However it has also caused the memory leak to start. We've reverted the CAPS and are back to how we were. Two questions: 1) No matter what the REST consumer is doing, the MGR should not accumulate memory, especially as we can see that the REST TCP connections have wrapped up. Is there anything more we can do to diagnose this? 2) Setting "allow *" worked, but is there are better setting just to allow the "pg dump" call (in addition to profile mgr)? Thanks, Chris

1 0

unable to deploy ceph -- failed to read label for XXX No such file or directory
by Radoslav Bodó 17 Apr '23

17 Apr '23

hello, during basic experimentation I'm running into wierd situaltion when adding osd to test cluster. The test cluster is created as 3x XEN DomU Debian Bookworm (test1-3), 4x CPU, 8GB RAM, xvda root, xvbd swap, 4x xvdj,k,l,m 20GB (LVM volumes in Dom0, propagated via xen phy device) and cleaned with `wipefs -a` ``` apt-get install cephadm ceph-common cephadm bootstrap --mon-ip 10.0.0.101 ceph orch host add test2 ceph orch host add test3 ``` when adding OSDs the first host gets created OSDs as expected, but during creating OSDs on second host the output gets wierd, even when adding each device separately the output shows that `ceph orch` tries to create multiple osds at once ``` root@test1:~# for xxx in j k l m; do ceph orch daemon add osd test2:/dev/xvd$xxx; done Created osd(s) 0,1,2,3 on host 'test2' Created osd(s) 0,1 on host 'test2' Created osd(s) 2,3 on host 'test2' Created osd(s) 1 on host 'test2' ``` the syslog on test2 node shows an errors ``` 2023-04-16T20:57:02.528456+00:00 test2 bash[10426]: cephadm 2023-04-16T20:57:01.389951+0000 mgr.test1.ucudzp (mgr.14206) 1691 : cephadm [INF] Found duplicate OSDs: osd.0 in status running on test1, osd.0 in status error on test2 2023-04-16T20:57:02.528748+00:00 test2 bash[10426]: cephadm 2023-04-16T20:57:01.391346+0000 mgr.test1.ucudzp (mgr.14206) 1692 : cephadm [INF] Removing daemon osd.0 from test2 -- ports [] 2023-04-16T20:57:02.528943+00:00 test2 bash[10426]: cluster 2023-04-16T20:57:02.350564+0000 mon.test1 (mon.0) 743 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-04-16T20:57:17.972962+00:00 test2 bash[20098]: stderr: failed to read label for /dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d: (2) No such file or directory 2023-04-16T20:57:17.973064+00:00 test2 bash[20098]: stderr: 2023-04-16T20:57:17.962+0000 7fad2451c540 -1 bluestore(/dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d) _read_bdev_label failed to open /dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d: (2) No such file or directory 2023-04-16T20:57:17.973181+00:00 test2 bash[20098]: --> Failed to activate via lvm: command returned non-zero exit status: 1 2023-04-16T20:57:17.973278+00:00 test2 bash[20098]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-04-16T20:57:17.973368+00:00 test2 bash[20098]: --> Failed to activate any OSD(s) ``` the ceph and cephadm binaries are installed from debian bookworm ``` ii ceph-common 16.2.11+ds-2 amd64 common utilities to mount and interact with a ceph storage cluster ii cephadm 16.2.11+ds-2 amd64 utility to bootstrap ceph daemons with systemd and containers ``` management session script can be found at https://pastebin.com/raw/FiX7DMHS none of the googled symptoms helped me to understand why is this situation happening nor how to troubleshoot or debug the issues. I'd understand that the nodes are very log on RAM to get this experiment running, but the behavior does not really look like OOM issue. any idea would be appreciated thanks bodik

1 1

RADOSGW zone data-pool migration.
by Gaël THEROND 17 Apr '23

17 Apr '23

Hi everyone, quick question regarding radosgw zone data-pool. I’m currently planning to migrate an old data-pool that was created with inappropriate failure-domain to a newly created pool with appropriate failure-domain. If I’m doing something like: radosgw-admin zone modify —rgw-zone default —data-pool <new_pool> Will data from the old pool be migrated to the new one or do I need to do something else to migrate those data out of the old pool? I’ve read a lot of mail archive with peoples willing to do that but I can’t get a clear answer from those archives. I’m running on nautilus release of it ever help. Thanks a lot!

1 0

Radosgw-admin bucket list has duplicate objects
by mahnoosh shahidi 17 Apr '23

17 Apr '23

Hi, I observed duplicate object names in the result of the admin list bucket on 15.2.12 cluster. I used the following command and some of the object names in the result list appeared more than once. There is no versioning config for the bucket. radosgw-admin bucket list --allow-unordered --max-entries=200000000 > --bucket=my-bucket > my-bucket.list Does anyone have any opinion on this? Should I make a bug report for this? Best Regards, Mahnoosh

3 3

OSDs remain not in after update to v17
by Alexandre Becholey 17 Apr '23

17 Apr '23

Dear Ceph Users, I have a small ceph cluster for VMs on my local machine. It used to be installed with the system packages and I migrated it to docker following the documentation. It worked OK until I migrated from v16 to v17 a few months ago. Now the OSDs remain "not in" as shown in the status: # ceph -s cluster: id: abef2e91-cd07-4359-b457-f0f8dc753dfa health: HEALTH_WARN 6 stray daemon(s) not managed by cephadm 1 stray host(s) with 6 daemon(s) not managed by cephadm 2 devices (4 osds) down 4 osds down 1 host (4 osds) down 1 root (4 osds) down Reduced data availability: 129 pgs inactive services: mon: 1 daemons, quorum bjorn (age 8m) mgr: bjorn(active, since 8m) osd: 4 osds: 0 up (since 4w), 4 in (since 4w) data: pools: 2 pools, 129 pgs objects: 0 objects, 0 B usage: 1.8 TiB used, 1.8 TiB / 3.6 TiB avail pgs: 100.000% pgs unknown 129 unknown I can see some network communication between the OSDs and the monitor and the OSDs are running: # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f8fbe8177a63 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-2 6768ec871404 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-1 ff82f84504d5 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-0 4c89e50ce974 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-3 fe0b6089edda quay.io/ceph/ceph:v17 "/usr/bin/ceph-mon -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mon-bjorn f76ac9dcdd6d quay.io/ceph/ceph:v17 "/usr/bin/ceph-mgr -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mgr-bjorn However when I try to use any `ceph orch` commands, they hang. I can also see some blacklist on the OSDs: # ceph osd blocklist ls 10.99.0.13:6833/3770763474 2023-04-13T08:17:38.885128+0000 10.99.0.13:6832/3770763474 2023-04-13T08:17:38.885128+0000 10.99.0.13:0/2634718754 2023-04-13T08:17:38.885128+0000 10.99.0.13:0/1103315748 2023-04-13T08:17:38.885128+0000 listed 4 entries The first two entries correspond to the manager process. `ceph osd blocked-by` does not show anything. I think I might have forgotten to set the `ceph osd require-osd-release ...` because 14 is written in `/var/lib/ceph/<ID>/osd.?/require_osd_release`. If I try to do it now, the monitor hits an abort: debug 0> 2023-04-12T08:43:27.788+0000 7f0fcf2aa700 -1 *** Caught signal (Aborted) ** in thread 7f0fcf2aa700 thread_name:ms_dispatch ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f0fd94bbcf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0fdb5124e3] 5: /usr/lib64/ceph/libceph-common.so.2(+0x26a64f) [0x7f0fdb51264f] 6: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basi 7: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x38d) [0x562719cb127d] 8: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x17b) [0x562719cb18cb] 9: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x2ce) [0x562719c20ade] 10: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x1ebb) [0x562719ab9f6b] 11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x9f2) [0x562719abe152] 12: (Monitor::_ms_dispatch(Message*)+0x406) [0x562719abf066] 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x562719aef13d] 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7f0fdb78e0e8] 15: (DispatchQueue::entry()+0x50f) [0x7f0fdb78b52f] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0fdb8543b1] 17: /lib64/libpthread.so.0(+0x81ca) [0x7f0fd94b11ca] 18: clone() Any ideas on what is going on? Many thanks, Alexandre

3 4

CEPH Mirrors are lacking packages
by Burkhard Linke 17 Apr '23

17 Apr '23

Hi, at least eu.ceph.com and de.ceph.com are lacking packages for the pacific release. All package not start with "c" (e.g. librbd, librados, radosgw) are missing. Best regards, Burkhard Linke

3 2

Cannot add disks back after their OSDs were drained and removed from a cluster
by stillsmil＠gmail.com 17 Apr '23

17 Apr '23

I find that I cannot re-add a disk to a Ceph cluster after the OSD on the disk is removed. Ceph seems to know about the existence of these disks, but not about their "host:dev" information: ``` # ceph device ls DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY SAMSUNG_MZ7L37T6_S6KHNE0T100049 0% <-- should be host01:sda, was osd.0 SAMSUNG_MZ7L37T6_S6KHNE0T100050 host01:sdb osd.1 0% SAMSUNG_MZ7L37T6_S6KHNE0T100052 0% <-- should be host02:sda SAMSUNG_MZ7L37T6_S6KHNE0T100053 host01:sde osd.9 0% SAMSUNG_MZ7L37T6_S6KHNE0T100061 host01:sdf osd.11 0% SAMSUNG_MZ7L37T6_S6KHNE0T100062 0% SAMSUNG_MZ7L37T6_S6KHNE0T100063 host01:sdc osd.5 0% SAMSUNG_MZ7L37T6_S6KHNE0T100064 host01:sdg osd.13 0% SAMSUNG_MZ7L37T6_S6KHNE0T100065 0% SAMSUNG_MZ7L37T6_S6KHNE0T100066 host01:sdd osd.7 0% SAMSUNG_MZ7L37T6_S6KHNE0T100067 0% SAMSUNG_MZ7L37T6_S6KHNE0T100068 0% <-- should be host02:sdb SAMSUNG_MZ7L37T6_S6KHNE0T100069 0% SAMSUNG_MZ7L37T6_S6KHNE0T100070 0% SAMSUNG_MZ7L37T6_S6KHNE0T100071 0% SAMSUNG_MZ7L37T6_S6KHNE0T100072 host01:sdh osd.15 0% SAMSUNG_MZQL27T6HBLA-00B7C_S6CVNG0T321600 host03:nvme4n1 osd.20 0% ... obmitted ... SAMSUNG_MZQL27T6HBLA-00B7C_S6CVNG0T321608 host03:nvme8n1 osd.22 0% ``` For disk "SAMSUNG_MZ7L37T6_S6KHNE0T100049", the "HOST:DEV" field is empty, while I believe it should be "host01:sda", as I have confirmed by running `smartctl -i /dev/sda" on host01. I guess the missing information is the reason that OSDs cannot be created, either manually or automatically on these devices. I have tried: 1. `ceph orch daemon add osd host01:/dev/sda` Prints "Created no osd(s) on host host01; already created?" 2. `ceph orch apply osd --all-available-devices` adds nothing I arrived at such situation while testing if draining a host works: I drained host02, removed it and added it back via: ``` ceph orch host drain host02 ceph orch host rm host02 ceph orch host add host02 <internal_ip> --labels _admin ``` I am running Ceph 17.2.6 on ubuntu. Ceph was deployed via cephadm. FYI the relavent orchestrator spec for osd: ``` # ceph orch ls osd --export service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore ``` Any thoughts on what maybe wrong here? Is there a way that I can tell Ceph "you are wrong about the whereabouts of these disks, forget what you know and fetch disk information afresh"? Any help much appreciated!

1 0

Ceph mon status stuck at "probing"
by York Huang 17 Apr '23

17 Apr '23

Hi Ceph community, I had a small lab cluster of version Nautilus (ceph-ansible / containerized) which functions quite well. As part of the upgrade experiment, I replaced one of the mon with an Octopus one (containerized as well, the OS is purged before mon deployment), the daemon seems working but its status is always "probing", and "ceph -s" showed that mon is down and out of quorum. I checked and compared the output of "ceph daemon XXX mon_status" of all monitors, ensuring there is no network issue, the new mon seemed to have the correct monmap but always stucked in status "probing". I doubt if Ceph mon of different versions do co-exist. Any further steps suggest? Thanks in advanced.

1 0

Dead node (watcher) won't timeout on RBD
by Max Boone 16 Apr '23

16 Apr '23

After a critical node failure on my lab cluster, which won't come back up and is still down, the RBD objects are still being watched / mounted according to ceph. I can't shell to the node to rbd unbind them as the node is down. I am absolutely certain that nothing is using these images and they don't have snapshots either (and this IP is not even remotely close to the those of the monitors in the cluster). I blocked the IP usingceph osd blocklist add but after 30 minutes, they are still being watched. Them being watched (they are RWO ceph-csi volumes) prevents me from re-using them in the cluster. As far as I'm aware, ceph should remove the watchers after 30 minutes and they've been blocklisted for hours now.root@node0:~# rbd status kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff Watchers: watcher=10.0.0.103:0/992994811 client.1634081 cookie=139772597209280 root@node0:~# rbd snap list kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff root@node0:~# rbd info kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff rbd image 'csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff': size 10 GiB in 2560 objects order 22 (4 MiB objects) snapshot_count: 0 id: 4ff5353b865e1 block_name_prefix: rbd_data.4ff5353b865e1 format: 2 features: layering op_features: flags: create_timestamp: Fri Mar 31 14:46:51 2023 access_timestamp: Fri Mar 31 14:46:51 2023 modify_timestamp: Fri Mar 31 14:46:51 2023 root@node0:~# rados -p kubernetes listwatchers rbd_header.4ff5353b865e1 watcher=10.0.0.103:0/992994811 client.1634081 cookie=139772597209280 root@node0:~# ceph osd blocklist ls 10.0.0.103:0/0 2023-04-16T13:58:34.854232+0200 listed 1 entries root@node0:~# ceph daemon osd.0 config get osd_client_watch_timeout { "osd_client_watch_timeout": "30" } Is it possible to kick a watcher out manually, or is there not much I can do here besides shutting down the entire cluster (or OSDs) and getting them back up? If it is a bug, I'm happy to help figuring out it's root cause and see if I can help writing a fix. Cheers, Max.

2 1

Re: Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 15 Apr '23

15 Apr '23

It's funny, I had the EPEL repos added yesterday, and no joy. Left it alone for a day, and voila, packages exist. Must have forgotten to update or clear my cache or something? Thanks for the help!

1 0

Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 15 Apr '23

15 Apr '23

Hello! I'm trying to Install the ceph-common package on a Rocky Linux 9 box so that I can connect to our ceph cluster and mount user directories. I've added the ceph repo to yum.repos.d, but when I run `dnf install ceph-common`, I get the following error ``` [root@jet yum.repos.d]# dnf install ceph-common Last metadata expiration check: 0:33:06 ago on Fri 14 Apr 2023 11:29:26 AM EDT. Error: Problem: conflicting requests - nothing provides libthrift-0.14.0.so()(64bit) needed by ceph-common-2:17.2.6-0.el9.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) ``` I've looked far and wide for this libthrift package, but I can't figure out what repo to add to get it. Has anyone had success installing ceph on Rocky 9 and would be willing to share some guidance?

2 1

ceph/daemon stable tag
by Jonas Nemeikšis 15 Apr '23

15 Apr '23

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

2 6

Mysteriously dead OSD process
by J-P Methot 15 Apr '23

15 Apr '23

Hi, We currently use Ceph Pacific 16.2.10 deployed with Cephadm on this storage cluster. Last night, one of our OSD died. However, since its storage is a SSD, we ran hardware checks and found no issue with the SSD itself. However, if we try starting the service again, the container just crashes 1 second after booting up. If I look at the logs, there's no error. You can see the OSD starting up normally and then the last line before the crash is : debug 2023-04-05T18:32:57.433+0000 7f8078e0c700 1 osd.87 pg_epoch: 207175 pg[2.99s3( v 207174'218628609 (207134'218623666,207174'218628609] local-lis/les=207140/207141 n=38969 ec=41966/315 lis/c=207140/207049 les/c/f=207141/207050/0 sis=207175 pruub=11.464111328s) [5,228,217,NONE,17,25,167,114,158,178,159]/[5,228,217,87,17,25,167,114,158,178,159]p5(0) r=3 lpr=207175 pi=[207049,207175)/1 crt=207174'218628605 mlcod 0'0 remapped NOTIFY pruub 12054.601562500s@ mbc={}] state<Start>: transitioning to Stray I don't really see how this line could cause the OSD to crash. Systemd just writes : Stopping Ceph osd.83 for (uuid) What could cause this OSD to boot up and then suddenly die? Outside the ceph daemon logs and the systemd logs, is there another way I could gain more information? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

2 1

Re: Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 14 Apr '23

14 Apr '23

Oops, forgot to mention that I'm installing Ceph 17.2.6, preempting an upgrade of our cluster from 15.2.17 to 17.2.6.

1 0

17.2.6 Dashboard/RGW Signature Mismatch
by Chris Palmer 14 Apr '23

14 Apr '23

Hi I have 3 Ceph clusters, all configured similarly, which have been happy for some months on 17.2.5: 1. A test cluster 2. A small production cluster 3. A larger production cluster All are debian 11 built from packages - no cephadm. I upgraded (1) to 17.2.6 without any problems at all. In particular the Object Gateway sections of the dashboard work as usual. I then upgraded (2). Nothing seemed amiss, and everything seems to work except... when I try to access the Object Gateway sections of the dashboard I always get: *The Object Gateway Service is not configured* Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","RequestId":"tx0000022ba920e82ac4a9c-0064381' b'934-10e73385-default","HostId":"10e73385-default-default"}') (Just the RequestId changes each time). Before the upgrade it worked just fine. Other info: * RGW requests using awscli and rclone all work with normal RGW accounts. It just seems to be the dashboard that's died. * Just the one zonegroup, no multisite/replication * "radosgw-admin user info --uid=rgwadmin" gives the correct output with the right access_key & secret_key. The other fields are as in (1). * "ceph dashboard get-rgw-api-access-key/get-rgw-api-secret-key" both give the right values. The rgw logs from (2) which fails show: 2023-04-13T16:36:28.720+0100 7fcc7966a700 1 ====== starting new request req=0x7fcd88c10720 ===== 2023-04-13T16:36:28.720+0100 7fcc80e79700 1 req 8090309398268968541 0.000000000s op->ERRORHANDLER: err_no=-2027 new_err_no=-2027 2023-04-13T16:36:28.724+0100 7fcc80e79700 1 ====== req done req=0x7fcd88c10720 op status=0 http_status=403 latency=0.003999980s ====== 2023-04-13T16:36:28.724+0100 7fcc80e79700 1 beast: 0x7fcd88c10720: 192.168.xx.xx - - [13/Apr/2023:16:36:28.720 +0100] "GET /admin/metadata/user?myself HTTP/1.1" 403 134 - "python-requests/2.25.1" - latency=0.003999980s (Note this does not have rgwadmin as the user, and is always the same URL) Whereas the rgw logs from (1) which works show things like: 2023-04-13T15:44:19.396+0000 7f8478da1700 1 ====== starting new request req=0x7f86284f5720 ===== 2023-04-13T15:44:19.412+0000 7f8478da1700 1 ====== req done req=0x7f86284f5720 op status=0 http_status=200 latency=0.016000060s ====== 2023-04-13T15:44:19.412+0000 7f8478da1700 1 beast: 0x7f86284f5720: 10.xx.xx.xx - rgwadmin [13/Apr/2023:15:44:19.396 +0000] "GET /admin/realm?list HTTP/1.1" 200 31 - "python-requests/2.25.1" - latency=0.016000060s (Note this has rgwadmin as the user, and various URLs) The only thing I can see in the release notes that looks even vaguely related is https://github.com/ceph/ceph/pull/47547, but it doesn't seem likely. I am really stumped on this, with no idea what has gone wrong on (2), and what the difference is between (1) and (2). I'm not going to touch (3) until I have resolved this. Grateful for any help... And thanks for all the good work. Regards, Chris

3 4

Restrict user to an RBD image in a pool
by Work Ceph 14 Apr '23

14 Apr '23

Hello guys! Is it possible to restrict user access to a single image in an RBD pool? I know that I can use namespaces, so users can only see images with a given namespaces. However, these users will still be able to create new RBD images. Is it possible to somehow block users from creating RBD images and only work with the already existing ones?

2 1

Cephadm only scheduling, not orchestrating daemons
by Thomas Widhalm 14 Apr '23

14 Apr '23

Hi, As you might know, I have a problem with MDS not starting. During the investigation with your help I found another issue that might be related. I can plan to restart, redeploy, reconfigure services via cephadm or dashboard just as I want but services won't react. I only see the action to be scheduled but that's all. 2023-04-13T17:11:15.690698+0000 mgr.ceph04.qaexpv (mgr.74906907) 37184 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph05.pqxmvt 2023-04-13T17:11:20.746743+0000 mgr.ceph04.qaexpv (mgr.74906907) 37190 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph06.rrxmks 2023-04-13T17:11:24.971226+0000 mgr.ceph04.qaexpv (mgr.74906907) 37195 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph07.omdisd It's the same for other daemons/services. I changed placement rules, scheduled changes, failed mgr, even rebooted hosts. I even was desperate enough to delete files for services from hosts before rebooting hoping I could trigger a manual redeploy. All I see are the same MDS stuck in "error" state. I removed them via "ceph orch rm" but they are still there. When I reissue the command it fails saying that the service isn't there. "ceph orch ps" still lists them. mds.mds01.ceph03.xqwdjy ceph03 error 2d ago 2M - - <unknown> <unknown> <unknown> mds.mds01.ceph04.hcmvae ceph04 error 2d ago 2d - - <unknown> <unknown> <unknown> mds.mds01.ceph05.pqxmvt ceph05 error 2d ago 10M - - <unknown> <unknown> <unknown> mds.mds01.ceph06.rrxmks ceph06 error 2d ago 10w - - <unknown> <unknown> <unknown> mds.mds01.ceph07.omdisd ceph07 error 2d ago 3M - - <unknown> <unknown> <unknown> Any idea how I can get rid of them? Or redeploy them? Additionally I'm just in the middle of an upgrade. { "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Upgrade paused", "is_paused": true } I paused it on purpose to allow for manipulation of daemons. Cheers, Thomas

2 1

ceph 17.2.6 and iam roles (pr#48030)
by Christopher Durham 13 Apr '23

13 Apr '23

Hi, I see that this PR: https://github.com/ceph/ceph/pull/48030 made it into ceph 17.2.6, as per the change log at: https://docs.ceph.com/en/latest/releases/quincy/ That's great. But my scenario is as follows: I have two clusters set up as multisite. Because of the lack of replication for IAM roles, we have set things up so that roles on the primary 'manually' get replicated to the secondary site via a python script. Thus, if I create a role on the primary, add/delete users or buckets from said role, the role, including the AssumeRolePolicyDocument and policies, gets pushed to the replicated site. This has served us well for three years. With the advent of this fix, what should I do before I upgrade to 17.2.6 (currently on 17.2.5, rocky 8) I know that in my situation, roles of the same name have different RoleIDs on the two sites. What should I do before I upgrade? Possibilities that *could* happen if i dont rectify things as we upgrade: 1. The different RoleIDs lead to two roles of the same name on the replicated site, perhaps with the system unable to address/look at/modify either 2. Roles just don't get repiicated to the second site or other similar situations, all of which I want to avoid. Perhaps the safest thing to do is to remove all roles on the secondary site, upgrade, and then force a replication of roles (How would I *force* that for iAM roles if it is the correct answer?) Here is the original bug report: https://tracker.ceph.com/issues/57364 Thanks! -Chris

2 3

Live migrate RBD image with a client using it
by Work Ceph 13 Apr '23

13 Apr '23

Hello guys, We have been reading the docs, and trying to reproduce that process in our Ceph cluster. However, we always receive the following message: ``` librbd::Migration: prepare: image has watchers - not migrating rbd: preparing migration failed: (16) Device or resource busy ``` We tested with both RDB blocks mounted with kRBD, and with librbd via KVM/qEMU system. Both cases result in the same result. For KRBD, we understood that it is not supported right now, but for librbd it seems that it should be supported somehow. How do you guys handle those situations? We have the following use cases that might need an image to migrate between pools while the client is still consuming it. - RBD images that are consumed via the iSCSI gateways - RBD images mounted (rbd map) in hosts - RBD images used by KVM/Libvirt Does Ceph support a live migration of images between pools while the clients/consumers are still using those volumes? [1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/

5 5

RGW is slowly after the ops increase
by Louis Koo 13 Apr '23

13 Apr '23

check the ops for rgw: [root@node06 ceph]# ceph daemon /var/run/ceph/ceph-client.rgw.os.dsglczutvqsgowpz.a.13.93908447458760.asok objecter_requests| jq ".ops" | jq 'length' 8 list subdir with s5cmd: [root@node01 deeproute]# time ./s5cmd --endpoint-url=http://10.x.x.x:80 ls s3://mlp-data-warehouse/ads_prediction/ DIR prediction_scenes/ DIR test_pai/ real 0m1.125s user 0m0.007s sys 0m0.016s after the ops increase: [root@node06 ceph]# ceph daemon /var/run/ceph/ceph-client.rgw.os.dsglczutvqsgowpz.a.13.93908447458760.asok objecter_requests| jq ".ops" | jq 'length' 264 list subdir with s5cmd: [root@node01 deeproute]# time ./s5cmd --endpoint-url=http://10.x.x.x:80 ls s3://mlp-data-warehouse/ads_prediction/ DIR prediction_scenes/ DIR test_pai/ real 0m8.822s user 0m0.004s sys 0m0.019s and if the ops increase to more 2000, it needs more than 100s to list the subdir, why?

1 0

Osd crash, looks like something related to PG recovery.
by Louis Koo 13 Apr '23

13 Apr '23

{ "archived": "2023-04-13 02:23:50.948191", "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7f2ee8198ce0]", "pthread_kill()", "(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x48c) [0x563506e9934c]", "(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >)+0x23e) [0x563506e9973e]", "(PrimaryLogPG::scan_range(int, int, BackfillInterval*, ThreadPool::TPHandle&)+0x15a) [0x56350699a8da]", "(PrimaryLogPG::do_scan(boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x914) [0x56350699bd34]", "(PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x776) [0x56350699c826]", "(OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x563506823fc9]", "(ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x563506a82e78]", "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x5635068414c8]", "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x563506ebe2a4]", "(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563506ec1184]", "/lib64/libpthread.so.0(+0x81ca) [0x7f2ee818e1ca]", "clone()" ], "ceph_version": "16.2.10", "crash_id": "2023-04-12T16:53:45.988696Z_7e73aedd-3518-41f4-ae48-e5dbfe5750ec", "entity_name": "osd.74", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "1a2700ce6c68288739eb14ca1b2b5f49449c59a5baafbd1e71df3a4316e3bffe", "timestamp": "2023-04-12T16:53:45.988696Z", "utsname_hostname": "node03", "utsname_machine": "x86_64", "utsname_release": "3.10.0-1160.45.1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021" } releated issuse under rook: https://github.com/rook/rook/issues/11565

1 0

radosgw crash
by Louis Koo 13 Apr '23

13 Apr '23

{ "archived": "2023-04-09 01:22:40.755345", "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7f06dc1edce0]", "(boost::asio::detail::reactive_socket_service_base::start_op(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, int, boost::asio::detail::reactor_op*, bool, bool, bool)+0x126) [0x7f06e6cf84d6]", "(void boost::asio::detail::reactive_socket_service_base::async_receive<boost::asio::mutable_buffer, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> >(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, boost::asio::mutable_buffer const&, int, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >&, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> const&)+0x1fa) [0x7f06e6d0bd1a]", "(boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >::operator()(boost::system::error_code, unsigned long, bool)+0x17f) [0x7f06e6d39f8f]", "(boost::asio::detail::executor_op<boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x189) [0x7f06e6d3a7d9]", "(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x8d) [0x7f06e6d0793d]", "(void boost::asio::io_context::executor_type::dispatch<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, std::allocator<void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>&&, std::allocator<void> const&) const+0x9c) [0x7f06e6d07b3c]", "(void boost::asio::detail::strand_executor_service::dispatch<boost::asio::io_context::executor_type const, boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>, std::allocator<void> >(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl> const&, boost::asio::io_context::executor_type const&, boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>&&, std::allocator<void> const&)+0x2b6) [0x7f06e6d3a306]", "(boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffer, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x1a1) [0x7f06e6d3a571]", "(boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2) [0x7f06e6cfbad2]", "/lib64/libradosgw.so.2(+0x430376) [0x7f06e6cde376]", "/lib64/libstdc++.so.6(+0xc2ba3) [0x7f06db212ba3]", "/lib64/libpthread.so.0(+0x81ca) [0x7f06dc1e31ca]", "clone()" ], "ceph_version": "16.2.10", "crash_id": "2023-04-08T22:37:20.389262Z_88939939-522f-4b2c-a5fb-d4e49e9922a7", "entity_name": "client.rgw.os.dsglczutvqsgowpz.a", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "radosgw", "stack_sig": "2535cd0a26a2ffcc7ca223d416ebf3d4ea172eeec60026bb8b36b2c97ea787da", "timestamp": "2023-04-08T22:37:20.389262Z", "utsname_hostname": "node01", "utsname_machine": "x86_64", "utsname_release": "3.10.0-1160.45.1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021" }

1 0

rookcmd: failed to configure devices: failed to generate osd keyring: failed to get or create auth key for client.bootstrap-osd:
by knawaz＠juniper.net 13 Apr '23

13 Apr '23

I am deploying Ceph via ROOK on K8s Cluster with following version metric ceph-version=17.2.5-0 Ubuntu 20.04 Kernel 5.4.0-135-generic But getting following error, does ceph-version=17.2.5-0 tested with Ubuntu 20.04 running Kernel 5.4.0-135-generic. Where I can find compatibility metric? 2023-04-12 21:34:41.958118 I | cephclient: getting or creating ceph auth key "client.bootstrap-osd" 2023-04-12 21:34:41.958184 D | exec: Running command: ceph auth get-or-create-key client.bootstrap-osd mon allow profile bootstrap-osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json 2023-04-12 21:34:57.291895 C | rookcmd: failed to configure devices: failed to generate osd keyring: failed to get or create auth key for client.bootstrap-osd: failed get-or-create-key client.bootstrap-osd: exit status 1'

1 0

Upgrade from 17.2.5 to 17.2.6 stuck at MDS
by Thomas Widhalm 12 Apr '23

12 Apr '23

Hi, If you remember, I hit bug https://tracker.ceph.com/issues/58489 so I was very relieved when 17.2.6 was released and started to update immediately. But now I'm stuck again with my broken MDS. MDS won't get into up:active without the update but the update waits for them to get into up:active state. Seems like a deadlock / chicken-egg problem to me. Since I'm still relatively new to Ceph, could you help me? What I see when watching the update status: { "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Error: UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host ceph01 at addr (192.168.23.61)", "is_paused": false } (The offline host was one host that broke during the upgrade. I fixed that in the meantime and the update went on.) And in the log: 2023-04-10T19:23:48.750129+0000 mgr.ceph04.qaexpv [INF] Upgrade: Waiting for mds.mds01.ceph04.hcmvae to be up:active (currently up:replay) 2023-04-10T19:23:58.758141+0000 mgr.ceph04.qaexpv [WRN] Upgrade: No mds is up; continuing upgrade procedure to poke things in the right direction Please give me a hint what I can do. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

3 9

Pacific - not able to add more mons while setting up new cluster
by Boris Behrens 12 Apr '23

12 Apr '23

Hi, I am currently facing a rather strange issue and I have no idea where to look next. ceph.conf [global] fsid = ee52ffdf-5f03-4ff1-9c8d-1761a52d5c6a ms_bind_ipv6 = true ms_bind_ipv4 = false #mon_initial_members = 08f1ea96ef02, 08f1ea98b7c2, 08f1ea98ab3e mon_host = [fd00:2380:be45:43::10],[fd00:2380:be45:43::11],[fd00:2380:be45:43::12] auth_cluster_required = none auth_service_required = none auth_client_required = none public_network = fd00:2380:be45:43::/64 I can create the first mon with: monmaptool --create --add `hostname -s` fd00:2380:be45:43::10 --fsid ee52ffdf-5f03-4ff1-9c8d-1761a52d5c6a /tmp/monmap sudo -u ceph ceph-mon --cluster ceph --mkfs -i `hostname -s` --monmap /tmp/monmap systemctl enable --now ceph-mon@`hostname -s`.service systemctl enable --now ceph-mgr@`hostname -s`.service On the first host I can use the ceph command line as I would expect. Normally I then go to the next host and add it to the mons with ceph mon getmap -o /tmp/monmap sudo -u ceph ceph-mon --cluster ceph --mkfs -i `hostname -s` --monmap /tmp/monmap systemctl enable --now ceph-mon@`hostname -s`.service systemctl enable --now ceph-mgr@`hostname -s`.service But the ceph command hangs forever. Ping between the hosts work and when I telnet the remote mon I also get an answer. We do not have any authentication in place, and normally I would expect that it works this way. Any ideas what I can try? I had it working yesterday, but the restarted all nodes at the same time and then it just stopped and I never got it working again. Can it be switch config? I've added the debug output but it looks for as both systems communicate with each other but somehow it does not work: root@08f1ea98b7c2:~# ceph status --debug_ms=20 2023-04-12T15:55:29.253+0000 7f38bed9d700 2 Event(0x7f38c0069620 nevent=5000 time_id=1).set_owner center_id=0 owner=139881696843520 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).create_file_event create event started fd=4 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.add_event add event fd=4 cur_mask=0 add_mask=1 to 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).create_file_event create event end fd=4 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 stack operator() starting 2023-04-12T15:55:29.253+0000 7f38be59c700 2 Event(0x7f38c0060960 nevent=5000 time_id=1).set_owner center_id=1 owner=139881688450816 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=1).create_file_event create event started fd=7 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.add_event add event fd=7 cur_mask=0 add_mask=1 to 6 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=1).create_file_event create event end fd=7 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38be59c700 10 stack operator() starting 2023-04-12T15:55:29.253+0000 7f38bdd9b700 2 Event(0x7f38c0061950 nevent=5000 time_id=1).set_owner center_id=2 owner=139881680058112 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=1).create_file_event create event started fd=10 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.add_event add event fd=10 cur_mask=0 add_mask=1 to 9 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=1).create_file_event create event end fd=10 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 stack operator() starting 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- ready 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 Processor -- start 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- start start 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect v2:[fd00:2380:be45:43::12]:3300/0, creating connection and registering 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_NONE l=0)._connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect 0x7f38c0102340 v2:[fd00:2380:be45:43::12]:3300/0 v2:[fd00:2380:be45:43::12]:3300/0 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- connect_to v2:[fd00:2380:be45:43::12]:3300/0 new 0x7f38c0102340 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect v1:[fd00:2380:be45:43::10]:6789/0, creating connection and registering 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_NONE l=0)._connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect 0x7f38c0104dd0 v1:[fd00:2380:be45:43::10]:6789/0 v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- connect_to v1:[fd00:2380:be45:43::10]:6789/0 new 0x7f38c0104dd0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING l=0).process 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect v1:[fd00:2380:be45:43::12]:6789/0, creating connection and registering 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_NONE l=0)._connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect 0x7f38c0108c30 v1:[fd00:2380:be45:43::12]:6789/0 v1:[fd00:2380:be45:43::12]:6789/0 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- connect_to v1:[fd00:2380:be45:43::12]:6789/0 new 0x7f38c0108c30 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING l=0).process 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_keepalive 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).wakeup 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING l=0).process 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- --> v1:[fd00:2380:be45:43::10]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c010ca30 con 0x7f38c0104dd0 2023-04-12T15:55:29.253+0000 7f38bf59e700 15 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_message inline write is denied, reschedule m=0x7f38c010ca30 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_keepalive 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- --> v1:[fd00:2380:be45:43::12]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c010dca0 con 0x7f38c0108c30 2023-04-12T15:55:29.253+0000 7f38bf59e700 15 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_message inline write is denied, reschedule m=0x7f38c010dca0 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- --> v2:[fd00:2380:be45:43::12]:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c010eef0 con 0x7f38c0102340 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event started fd=12 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bf59e700 5 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).send_message enqueueing message m=0x7f38c010eef0 type=5 mon_getmap magic: 0 v1 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=0 add_mask=1 to 6 2023-04-12T15:55:29.253+0000 7f38bf59e700 15 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).send_message inline write is denied, reschedule m=0x7f38c010eef0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event end fd=12 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38be59c700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event started fd=13 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38be59c700 10 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process nonblock connect inprogress 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event started fd=12 mask=2 original mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.add_event add event fd=13 cur_mask=0 add_mask=1 to 3 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=1 add_mask=2 to 6 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event end fd=13 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event end fd=12 mask=2 current mask is 3 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event started fd=14 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.add_event add event fd=14 cur_mask=0 add_mask=1 to 9 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process nonblock connect inprogress 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event end fd=14 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event started fd=13 mask=2 original mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.add_event add event fd=13 cur_mask=1 add_mask=2 to 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event end fd=13 mask=2 current mask is 3 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).handle_write 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).write_event 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).process nonblock connect inprogress 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event started fd=14 mask=2 original mask is 1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.add_event add event fd=14 cur_mask=1 add_mask=2 to 9 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 NetHandler reconnect reconnect: 111 Connection refused 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event end fd=14 mask=2 current mask is 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 1 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:[fd00:2380:be45:43::12]:6789/0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process 2023-04-12T15:55:29.253+0000 7f38bed9d700 2 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process connection refused! 2023-04-12T15:55:29.253+0000 7f38be59c700 10 NetHandler reconnect reconnect: 111 Connection refused 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).handle_write 2023-04-12T15:55:29.253+0000 7f38be59c700 1 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:[fd00:2380:be45:43::12]:3300/0 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).fault 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).write_event 2023-04-12T15:55:29.253+0000 7f38be59c700 2 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process connection refused! 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.del_event del event fd=13 cur_mask=3 delmask=3 to 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 5 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state 2023-04-12T15:55:29.253+0000 7f38bed9d700 5 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).fault waiting 0.200000 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.del_event del event fd=14 cur_mask=3 delmask=2 to 9 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).process connect successfully, ready to send banner 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).read_event 2023-04-12T15:55:29.253+0000 7f38be59c700 10 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_client_banner 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.del_event del event fd=12 cur_mask=3 delmask=3 to 6 2023-04-12T15:55:29.253+0000 7f38be59c700 5 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_recv_state 2023-04-12T15:55:29.253+0000 7f38be59c700 5 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_security 2023-04-12T15:55:29.253+0000 7f38be59c700 1 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 9 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING pgs=0 cs=0 l=0).handle_client_banner_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING pgs=0 cs=0 l=0).handle_client_banner_write connect write banner done: v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).wait_server_banner 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=281 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=281 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify connect read peer addr v1:[fd00:2380:be45:43::10]:6789/0 on socket 14 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify connect peer addr for me is v1:[fd00:2380:be45:43::11]:44160/0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 1 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:[fd00:2380:be45:43::10]:6789/0 says I am v1:[fd00:2380:be45:43::11]:44160/0 (socket says [fd00:2380:be45:43::11]:44160) 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 learned_addr had no addrs 2023-04-12T15:55:29.253+0000 7f38bdd9b700 1 -- [fd00:2380:be45:43::11]:0/2007947341 learned_addr learned my addr [fd00:2380:be45:43::11]:0/2007947341 (peer_addr_for_me v1:[fd00:2380:be45:43::11]:0/0) 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 136 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_my_addr_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_my_addr_write connect sent my addr v1:[fd00:2380:be45:43::11]:0/2007947341 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).send_connect_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).send_connect_message connect sending gseq=1 cseq=0 proto=15 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 33 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_message_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_message_write connect wrote (self +) cseq, waiting for reply 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).wait_connect_reply 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=26 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=26 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 connect got reply tag 13 connect_seq 1 global_seq 52 proto 15 flags 1 features 4540138314316775423 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 got CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).wait_ack_seq 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=8 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_ack_seq r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_ack_seq got newly_acked_seq 0 vs out_seq 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).discard_requeued_up_to 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 8 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_in_seq_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_in_seq_write send in_seq done 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).client_ready 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=52 cs=1 l=1).client_ready connect success 1, lossy = 1, features 4540138314316775423 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=52 cs=1 l=1).client_ready no authorizer, clearing session_security 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_event 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message m auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c010ca30 auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending message type=17 src client.? front=60 data=0 off 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 1 0x7f38c010ca30 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 144 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 0x7f38c010ca30 done. 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 15 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_keepalive2_ack r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_keepalive2_ack got KEEPALIVE_ACK 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=18 src mon.0 front=24 data=0 off 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=24 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 24 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 24 + 0 + 0 byte message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.253+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 1 0x7f38c010ca30 auth_reply(proto 1 0 (0) Success) v1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38c010ca30 prio 196 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.253+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 1 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (unknown 1154771820 0 0) 0x7f38c010ca30 con 0x7f38c0104dd0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.257+0000 7f38bd59a700 5 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).discard_out_queue discard mon_getmap magic: 0 v1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.257+0000 7f38be59c700 5 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_recv_state (warped) reseting crypto handlers 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-04-12T15:55:29.257+0000 7f38be59c700 5 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_security 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 2 0x7f38b4014b00 mon_map magic: 0 v1 2023-04-12T15:55:29.257+0000 7f38bd59a700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38b4014b00 prio 196 2023-04-12T15:55:29.257+0000 7f38bd59a700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).discard_out_queue discard 0x7f38c010dca0 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c010f080 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bd59a700 15 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c010f080 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 24 to dispatch throttler 239/104857600 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38c010ca30 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_event 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 2 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b4014b00 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message m mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c010f080 mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending message type=15 src client.? front=58 data=0 off 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 2 0x7f38c010f080 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 133 remaining bytes 0 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38b4014b00 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 0x7f38c010f080 done. 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=62 src mon.0 front=4 data=0 off 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=4 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 4 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 4 + 0 + 0 byte message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 3 0x7f38b4014df0 config(0 keys) v1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38b4014df0 prio 196 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 3 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b4014df0 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 4 to dispatch throttler 4/104857600 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38b4014df0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 4 0x7f38b40151a0 mon_map magic: 0 v1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38b40151a0 prio 196 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 4 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b40151a0 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38b40151a0 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CLOSED pgs=52 cs=1 l=1).reset_recv_state (warped) reseting security handlers 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CLOSED pgs=52 cs=1 l=1).reset_security 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 EpollDriver.del_event del event fd=14 cur_mask=1 delmask=3 to 9 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown [fd00:2380:be45:43::11]:0/2007947341 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 Processor -- stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections mark down v1:[fd00:2380:be45:43::12]:6789/0 0x7f38c0108c30 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=CLOSED pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections mark down v1:[fd00:2380:be45:43::10]:6789/0 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CLOSED pgs=52 cs=1 l=1).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections mark down v2:[fd00:2380:be45:43::12]:3300/0 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0108c30 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 msgr2=0x7f38c00632e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 wait: waiting for dispatch queue 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 wait: dispatch queue is stopped 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0062e70 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 wait: done. 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 wait complete. 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- ready 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 Processor -- start 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- start start 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- create_connect v1:[fd00:2380:be45:43::10]:6789/0, creating connection and registering 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_NONE l=1)._connect 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- create_connect 0x7f38c0102340 v1:[fd00:2380:be45:43::10]:6789/0 v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- connect_to v1:[fd00:2380:be45:43::10]:6789/0 new 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).send_keepalive 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- --> v1:[fd00:2380:be45:43::10]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c0067e30 con 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 15 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f38c0067e30 2023-04-12T15:55:29.257+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING l=1).process 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event started fd=12 mask=1 original mask is 0 2023-04-12T15:55:29.257+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=0 add_mask=1 to 6 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event end fd=12 mask=1 current mask is 1 2023-04-12T15:55:29.257+0000 7f38be59c700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.257+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).process nonblock connect inprogress 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event started fd=12 mask=2 original mask is 1 2023-04-12T15:55:29.257+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=1 add_mask=2 to 6 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event end fd=12 mask=2 current mask is 3 2023-04-12T15:55:29.257+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).handle_write 2023-04-12T15:55:29.257+0000 7f38be59c700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).write_event 2023-04-12T15:55:29.257+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).process 2023-04-12T15:55:29.257+0000 7f38be59c700 20 EpollDriver.del_event del event fd=12 cur_mask=3 delmask=2 to 6 2023-04-12T15:55:29.257+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).process connect successfully, ready to send banner 2023-04-12T15:55:29.257+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).read_event 2023-04-12T15:55:29.257+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).send_client_banner 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 9 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING pgs=0 cs=0 l=1).handle_client_banner_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING pgs=0 cs=0 l=1).handle_client_banner_write connect write banner done: v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).wait_server_banner 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=281 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=281 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify connect read peer addr v1:[fd00:2380:be45:43::10]:6789/0 on socket 12 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify connect peer addr for me is v1:[fd00:2380:be45:43::11]:44170/0 2023-04-12T15:55:29.261+0000 7f38be59c700 1 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:[fd00:2380:be45:43::10]:6789/0 says I am v1:[fd00:2380:be45:43::11]:44170/0 (socket says [fd00:2380:be45:43::11]:44170) 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 learned_addr had no addrs 2023-04-12T15:55:29.261+0000 7f38be59c700 1 -- [fd00:2380:be45:43::11]:0/4275451039 learned_addr learned my addr [fd00:2380:be45:43::11]:0/4275451039 (peer_addr_for_me v1:[fd00:2380:be45:43::11]:0/0) 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 136 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_my_addr_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_my_addr_write connect sent my addr v1:[fd00:2380:be45:43::11]:0/4275451039 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).send_connect_message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).send_connect_message connect sending gseq=1 cseq=0 proto=15 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 33 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_message_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_message_write connect wrote (self +) cseq, waiting for reply 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).wait_connect_reply 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=26 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=26 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 connect got reply tag 13 connect_seq 1 global_seq 53 proto 15 flags 1 features 4540138314316775423 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 got CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).wait_ack_seq 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_ack_seq r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 2 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_ack_seq got newly_acked_seq 0 vs out_seq 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).discard_requeued_up_to 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 8 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_in_seq_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_in_seq_write send in_seq done 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).client_ready 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=53 cs=1 l=1).client_ready connect success 1, lossy = 1, features 4540138314316775423 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=53 cs=1 l=1).client_ready no authorizer, clearing session_security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c0067e30 auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=17 src client.? front=60 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 1 0x7f38c0067e30 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 144 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c0067e30 done. 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 15 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_keepalive2_ack r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_keepalive2_ack got KEEPALIVE_ACK 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=18 src mon.0 front=24 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=24 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 24 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 24 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 1 0x7f38c0067e30 auth_reply(proto 1 0 (0) Success) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38c0067e30 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 1 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (unknown 2435360628 0 0) 0x7f38c0067e30 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0068070 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38aeffd700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c0068070 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 24 to dispatch throttler 239/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38c0067e30 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 2 0x7f38b0025be0 mon_map magic: 0 v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38b0025be0 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 2 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b0025be0 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c0068070 mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38b0025be0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=15 src client.? front=58 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 2 0x7f38c0068070 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 133 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c0068070 done. 2023-04-12T15:55:29.261+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c0068350 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38bf59e700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c0068350 2023-04-12T15:55:29.261+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_subscribe({mgrmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c0068350 mon_subscribe({mgrmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=15 src client.4108 front=39 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 3 0x7f38c0068350 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 114 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c006b410 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c0068350 done. 2023-04-12T15:55:29.261+0000 7f38bf59e700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c006b410 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_subscribe({osdmap=0}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c006b410 mon_subscribe({osdmap=0}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=15 src client.4108 front=39 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 4 0x7f38c006b410 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 114 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c006b410 done. 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=62 src mon.0 front=4 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=4 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 4 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 4 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 3 0x7f38b0025f40 config(0 keys) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38b0025f40 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 3 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b0025f40 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 4 to dispatch throttler 4/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38b0025f40 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 4 0x7f38b0026270 mon_map magic: 0 v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38b0026270 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 4 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b0026270 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38b0026270 2023-04-12T15:55:29.261+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c004f50 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38bf59e700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f389c004f50 2023-04-12T15:55:29.261+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_command({"prefix": "get_command_descriptions"} v 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f389c004f50 mon_command({"prefix": "get_command_descriptions"} v 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=50 src client.4108 front=80 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 5 0x7f389c004f50 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 155 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f389c004f50 done. 2023-04-12T15:55:32.261+0000 7f38adffb700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_keepalive 2023-04-12T15:55:32.261+0000 7f38adffb700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:32.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:32.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:32.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:32.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 9 remaining bytes 0 2023-04-12T15:55:42.261+0000 7f38adffb700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_keepalive 2023-04-12T15:55:42.261+0000 7f38adffb700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:42.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:42.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:42.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:42.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 9 remaining bytes 0 ^CInterrupted -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 1

pacific v16.2.1 (hot-fix) QE Validation status
by Yuri Weinstein 12 Apr '23

12 Apr '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59426#note-3 Release Notes - TBD Seeking approvals/reviews for: smoke - Josh approved? orch - Adam King approved? (there are infrastructure issues in the runs, but we want to release this ASAP) Thx YuriW

3 2

RBD snapshot mirror syncs all snapshots
by Andreas Teuchert 12 Apr '23

12 Apr '23

Hello, I setup two-way snapshot-based RBD mirroring between two Ceph clusters. After enabling mirroring for an image that already had regular snapshots independently from RBD mirror on the source cluster, the image and all snapshots were synced to the destination cluster. Is there a way to avoid having all snapshots being synced? We only need the latest version of the image on the destination cluster and the snapshots add around 200% disk space overhead on average. Best regards, Andreas

1 0

Ceph Leadership Team Meeting, 2023-04-12 Minutes
by Patrick Donnelly 12 Apr '23

12 Apr '23

Hi folks, Today we discussed: - Just short of 1 exabyte of Ceph storage reported to Telemetry. Telemetry's data is public and viewable at: https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?orgId=1 If your cluster is not reporting to Telemetry, please consider it! :) - A request from the Ceph Foundation Board to begin tracking component (e.g. CephFS) roadmaps in docs (or somewhere else appropriate). Concurrently, leads may also begin sending out status updates on a ~quarterly basis. To be discussed further. - Cephalocon schedule is available: https://ceph2023.sched.com/ - A regression was reported for the exporter in 17.2.6: https://github.com/ceph/ceph/pull/50718#issuecomment-1503376925 A follow-up hotfix/announcement is planned. - Next week's meeting is canceled due to Cephalocon/travel. Meeting minutes available here as always: https://pad.ceph.com/p/clt-weekly-minutes -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: avg apply latency went up after update from octopus to pacific
by Jan-Tristan Kruse 12 Apr '23

12 Apr '23

Hi, we solved our latency-problems in two clusters by redeploying all OSDs. Since then, we did not reencounter the problems described. Greetings, Jan

2 1

[RGW] Rebuilding a non master zone
by Gilles Mocellin 12 Apr '23

12 Apr '23

Hello cephers ! As I was asking in another thread ([RGW] Rebuilding a non master zone), I try to find the best way to rebuild a zone in a multisite config. The goal is to get rid of remaining Large OMAP objects. The simplest way, as I can rely only on the primary zone, is to : - remove the zone from the zonegroup - delete the pools - recreate them by restarting radosgw instances - add the zone in the zonegroup Is there a simpler way ? One question I already asked was : can I delete only index and data pools (ZONE.rgw.buckets.data and ZONE.rgw.buckets.index), the multisite config (zones and zonegroup) should be still there, and perhaps it will just resync (or I can force a sync init) ? Another thing, about ceph-ansible, which I still use. I configure my zones with the multisite config of ceph-ansible, but one thing bothers me : it defines the zone endpoints on each radosgw instances, even with the loadbalancer configured. So, do you think I'd better not configure multisite with ceph-ansible, and do it manually ? Is there a way that will make things easier to migrate to cephadm ?

1 0

Pacific dashboard: unable to get RGW information
by Michel Jouvin 12 Apr '23

12 Apr '23

Hi, Our cluster is running Pacific 16.2.10. We have a problem using the dashboard to display information about RGWs configured in the cluster. When clicking on "Object Gateway", we get an error 500. Looking in the mgr logs, I found that the problem is that the RGW is accessed by its IP address rather than its name. As the RGW has SSL enabled, the certificate cannot be matched against the IP address. I digged into the configuration but I was not able to identify where an IP address rather than a name was used (I checked in particular the zonegroup parameters and names are used to define endpoints). Did I make something wrong in the configuration or is it a know issue when using SSL-enabled RGW? Best regards, Michel

4 7

Module 'cephadm' has failed: invalid literal for int() with base 10:
by Duncan M Tooke 12 Apr '23

12 Apr '23

Hi, Our Ceph cluster is in an error state with the message: # ceph status cluster: id: 58140ed2-4ed4-11ed-b4db-5c6f69756a60 health: HEALTH_ERR Module 'cephadm' has failed: invalid literal for int() with base 10: '352.broken' This happened after trying to re-add an OSD which had failed. Adopting it back in to the Ceph failed because a directory was causing problems in /var/lib/ceph/{cephid}/osd.352. To re-add the OSD I renamed it to osd.352.broken (rather than delete it), re-ran the command and then everything worked perfectly. Then 5 minutes later the ceph orchestrator went into "HEALTH_ERR" I've removed that directory, but "cephadm" isn't cleaning up after itself. Does anyone know if there's a way I can clear the cache for this directory it's tried to inventory and failed? Thanks, Duncan -- Dr Duncan Tooke | Research Cluster Administrator Centre for Computational Biology, Weatherall Institute of Molecular Medicine, University of Oxford, OX3 9DS www.imm.ox.ac.uk<http://www.imm.ox.ac.uk>

2 2

radosgw-admin bucket stats doesn't show real num_objects and size
by viplanghe6＠gmail.com 12 Apr '23

12 Apr '23

The radosgw-admin bucket stats show there are 209266 objects in this bucket, but it included failed multiparts, so that make the size parameter is also wrong. When I use boto3 to count objects, the bucket only has 209049 objects. The only solution I can find is to use lifecycle to clean these failed multiparts, but in production, the client will decide to use lifecycle or not? So are there any way to exclude the failed multiparts in bucket statistic? Does Ceph allow to set auto clean failed multiparts globally? Thanks! "usage": { "rgw.main": { "size": 593286801276, "size_actual": 593716080640, "size_utilized": 593286801276, "size_kb": 579381642, "size_kb_actual": 579800860, "size_kb_utilized": 579381642, "num_objects": 209266 }

3 2

naming the S release
by Josh Durgin 11 Apr '23

11 Apr '23

With the Reef dev cycle closing, it's time to think about S and future releases. There are a bunch of options for S already, add a +1 or a new option to this etherpad, and we'll see what has the most votes next week: https://pad.ceph.com/p/s Josh

1 0

Ceph Object Gateway and lua scripts
by Thomas Bennett 11 Apr '23

11 Apr '23

Hi, We're currently testing out lua scripting in the Ceph Object Gateway (Radosgw). Ceph version: 17.2.5 We've tried a simple experiment with the simple lua script which is based on the documentation (see fixed width text below). However, the issue we're having is that we can't find the log messages anywhere. We've searched the entire jourrnalctl database as well as raised the debug level on the radosgw by setting debug_rgw to 20 on the running daemon. Any help welcome :) function print_object(msg, object) RGWDebugLog(" Title: " .. msg) RGWDebugLog(" Name: " .. object.Name) RGWDebugLog(" Instance: " .. object.Instance) RGWDebugLog(" Id: " .. object.Id) RGWDebugLog(" Size: " .. object.Size) RGWDebugLog(" MTime: " .. object.MTime) end RGWDebugLog("This is a log message!") Request.Log() if Request.CopyFrom then print_object("copy from", Request.CopyFrom.Object) if Request.CopyFrom.Object then print_object("copy from-object" ,Request.CopyFrom.Object) end end if Request.Object then print_object("Object" ,Request.Object) end Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.

2 3

RGW don't use .rgw.root multisite configuration
by Guillaume Morin 11 Apr '23

11 Apr '23

Hello, for my multisite configuration, i create and use an root pool other than .rgw.root to store realm and zone configuration with the follow option: rgw_realm_root_pool=myzone.rgw.root rgw_zonegroup_root_pool=myzone.rgw.root rgw_zone_root_pool=myzone.rgw.root But , i can see during the syncronize with my secondary zone (real only) , radosgw need to read and write period in .rgw.root. Is there an option or other to avoid read and write on the root pool .rgw.root ? Is it mandatory ? Pacific regards Guillaume

3 2

Announcing go-ceph v0.21.0
by John Mulligan 11 Apr '23

11 Apr '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.21.0 Changes include additions to the rbd, cephfs, and cephfs/admin packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Why is my cephfs almostfull?
by Jorge Garcia 11 Apr '23

11 Apr '23

We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space: data: pools: 3 pools, 4104 pgs objects: 1.09 G objects, 944 TiB usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.5 PiB 1.0 PiB 1.5 PiB 59.76 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 2 944 TiB 87.63 133 TiB 880988429 cephfs_metadata 3 128 MiB 0 62 TiB 206535313 .rgw.root 4 0 B 0 62 TiB 0 The whole thing consists of 2 pools: metadata (regular default replication) and data (erasure k:5 m:2). The global raw space reports 2.5 PiB total, with 1.0 PiB still available. But, when the ceph filesystem is mounted, it only reports 1.1 PB total, and the filesystem is almost full: Filesystem Size Used Avail Use% Mounted on x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups So, where is the rest of my space? Or what am I missing? Thanks!

3 2

Disks are filling up even if there is not a single placement group on them
by Michal Strnad 11 Apr '23

11 Apr '23

Hi all, Our SSD disks are filling up, even if there is not a single placement group on them. How is that possible please? ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1997 ssd 1.15269 1.00000 1.2 TiB 943 GiB 942 GiB 6 KiB 1.5 GiB 237 GiB 79.90 1.71 0 up Cluster uses Pacific 16.2.7. Thank you Michal

3 4

ceph.v17 multi-mds ephemeral directory pinning: cannot set or retrieve extended attribute
by Ulrich Pralle 10 Apr '23

10 Apr '23

Hi, we are using ceph version 17.2.5 on Ubuntu 22.04.1 LTS. We deployed multi-mds (max_mds=4, plus standby-replay mds). Currently we statically directory-pinned our user home directories (~50k). The cephfs' root directory is pinned to '-1', ./homes is pinned to "0". All user home directories below ./homes/ are pinned to -1, 1, 2, or 3 depending on a simple hash algorithm. Cephfs is provided to our users as samba/cifs (clustered samba,ctdb). We want to try ephemeral directory pinning. We can successfully set the extended attribute "ceph.dir.pin.distributed" with setfattr(1), but cannot retrieve its setting afterwards.: # setfattr -n ceph.dir.pin.distributed -v 1 ./units # getfattr -n ceph.dir.pin.distributed ./units ./units: ceph.dir.pin.distributed: No such attribute strace setfattr reports success on setxattr setxattr("./units", "ceph.dir.pin.distributed", "1", 1, 0) = 0 strace getfattr reports lstat("./units", {st_mode=S_IFDIR|0751, st_size=1, ...}) = 0 getxattr("./units", "ceph.dir.pin.distributed", NULL, 0) = -1 ENODATA (No data available) The file system is mounted rw,noatime,,name=<omitted>,mds_namespace=<omitted>.acl,recover_session=clean. The cephfs mds caps are "allow rwps". "./units" has a ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=fs_data_units" Ubuntu's setfattr is version 2.4.48. Defining other cephfs extend attributes (like ceph.dir.pin, ceph.quota.max_bytes, etc.) works as expected. What are we missing? Should we clear all static directory pinnings in advance? Are there any experience with ephemeral directory pinning? Or should one refrain from multi-mds at all? Kind regards Uli Pralle

2 1

v17.2.6 Quincy released
by Yuri Weinstein 10 Apr '23

10 Apr '23

We're happy to announce the 6th backport release in the Quincy series. https://ceph.io/en/news/blog/2023/v17-2-6-quincy-released/ Notable Changes --------------- * `ceph mgr dump` command now outputs `last_failure_osd_epoch` and `active_clients` fields at the top level. Previously, these fields were output under `always_on_modules` field. * telemetry: Added new metrics to the 'basic' channel to report per-pool bluestore compression metrics. See a sample report with `ceph telemetry preview`. Opt-in with `ceph telemetry on`. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.6.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: d7ff0d10654d2280e08f1ab989c7cdf3064446a5

1 0

quincy v17.2.6 QE Validation status
by Yuri Weinstein 07 Apr '23

07 Apr '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59070#note-1 Release Notes - TBD The reruns were in the queue for 4 days because of some slowness issues. The core team (Neha, Radek, Laura, and others) are trying to narrow down the root cause. Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to test and merge at least one PR https://github.com/ceph/ceph/pull/50575 for the core) rgw - Casey fs - Venky (the fs suite has an unusually high amount of failed jobs, any reason to suspect it in the observed slowness?) orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - Laura is looking into failures upgrade/pacific-x - Laura is looking into failures upgrade/quincy-p2p - Laura is looking into failures client-upgrade-octopus-quincy-quincy - missing packages, Adam Kraitman is looking into it powercycle - Brad ceph-volume - needs a rerun on merged https://github.com/ceph/ceph-ansible/pull/7409 Please reply to this email with approval and/or trackers of known issues/PRs to address them. Also, share any findings or hypnosis about the slowness in the execution of the suite. Josh, Neha - gibba and LRC upgrades pending major suites approvals. RC release - pending major suites approvals. Thx YuriW

14 27

Cephadm - Error ENOENT: Module not found
by elia.oggian＠cscs.ch 06 Apr '23

06 Apr '23

Hello, After a successful upgrade of a Ceph cluster from 16.2.7 to 16.2.11, I needed to downgrade it back to 16.2.7 as I found an issue with the new version. I expected that running the downgrade with:`ceph orch upgrade start --ceph-version 16.2.7` should have worked fine. However, it blocked right after the downgrade of the first MGR daemon. In fact, the downgraded daemon is not able to use the cephadm module anymore. Any `ceph orch` command fails with the following error: ``` $ ceph orch ps Error ENOENT: Module not found ``` And the downgrade process is therefore blocked. These are the logs of the MGR when issuing the command: ``` Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.557+0000 7f828fe8c700 0 log_channel(audit) log [DBG] : from='client.3136173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.list_daemons(*(None, None), **{'daemon_id': None, 'host': None, 'refresh': False}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.get_feature_set(*(), **{}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 mgr.server reply reply (2) No such file or directory Module not found ``` Other interesting MGR logs are: ``` 2023-03-28T11:05:59.519+0000 7fcd16314700 4 mgr get_store get_store key: mgr/cephadm/upgrade_state 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Failed to construct class in 'cephadm' 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Traceback (most recent call last): e "/usr/share/ceph/mgr/cephadm/module.py", line 450, in __init__ elf.upgrade = CephadmUpgrade(self) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 111, in __init__ elf.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 92, in from_json eturn cls(**c) rror: __init__() got an unexpected keyword argument 'daemon_types' 2023-03-28T11:05:59.521+0000 7fcd16314700 -1 mgr operator() Failed to run module in active mode ('cephadm') ``` Which seem to relate to the new feature of staggered upgrades. Please note that before, everything was working fine with version 16.2.7. I am currently stuck in this situation with only one MGR daemon on version 16.2.11 which is the only one still working fine: ``` [root@astano01 ~]# ceph orch ps | grep mgr mgr.astano02.mzmewn astano02 *:8443,9283 running (5d) 43s ago 2y 455M - 16.2.11 7a63bce27215 e2d7806acf16 mgr.astano03.qtzccn astano03 *:8443,9283 running (3m) 22s ago 95m 383M - 16.2.7 463ec4b1fdc0 cc0d88864fa1 ``` Does anyone already faced this issue or knows how can I make the 16.2.7 MGR load the cephadm module correctly? Thanks in advance for any help!

2 2

Misplaced objects greater than 100%
by Johan Hattne 06 Apr '23

06 Apr '23

Dear all; Up until a few hours ago, I had a seemingly normally-behaving cluster (Quincy, 17.2.5) with 36 OSDs, evenly distributed across 3 of its 6 nodes. The cluster is only used for CephFS and the only non-standard configuration I can think of is that I had 2 active MDSs, but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB (all OSD hosts have 256 G of RAM) at some point in the past. Then I rebooted one of the OSD nodes. The rebooted node held one of the active MDSs. Now the node is back up: ceph -s says the cluster is healthy, but all PGs are in a active+clean+remapped state and 166.67% of the objects are misplaced (dashboard: -66.66% healthy). The data pool is a threefold replica with 5.4M object, the number of misplaced objects is reported as 27087410/16252446. The denominator in the ratio makes sense to me (16.2M / 3 = 5.4M), but the numerator does not. I also note that the ratio is *exactly* 5 / 3. The filesystem is still mounted and appears to be usable, but df reports it as 100% full; I suspect it would say 167% but that is capped somewhere. Any ideas about what is going on? Any suggestions for recovery? // Best wishes; Johan

3 6

CephFS thrashing through the page cache
by Ashu Pachauri 05 Apr '23

05 Apr '23

We have an internal use case where we back the storage of a proprietary database by a shared file system. We noticed something very odd when testing some workload with a local block device backed file system vs cephfs. We noticed that the amount of network IO done by cephfs is almost double compared to the IO done in case of a local file system backed by an attached block device. We also noticed that CephFS thrashes through the page cache very quickly compared to the amount of data being read and think that the two issues might be related. So, I wrote a simple test. 1. I wrote 10k files 400KB each using dd (approx 4 GB data). 2. I dropped the page cache completely. 3. I then read these files serially, again using dd. The page cache usage shot up to 39 GB for reading such a small amount of data. Following is the code used to repro this in bash: for i in $(seq 1 10000); do dd if=/dev/zero of=test_${i} bs=4k count=100 done sync; echo 1 > /proc/sys/vm/drop_caches for i in $(seq 1 10000); do dd if=test_${i} of=/dev/null bs=4k count=100 done The ceph version being used is: ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) The ceph configs being overriden: WHO MASK LEVEL OPTION VALUE RO mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/mode upmap mgr advanced mgr/dashboard/server_addr 127.0.0.1 * mgr advanced mgr/dashboard/server_port 8443 * mgr advanced mgr/dashboard/ssl false * mgr advanced mgr/prometheus/server_addr 0.0.0.0 * mgr advanced mgr/prometheus/server_port 9283 * osd advanced bluestore_compression_algorithm lz4 osd advanced bluestore_compression_mode aggressive osd advanced bluestore_throttle_bytes 536870912 osd advanced osd_max_backfills 3 osd advanced osd_op_num_threads_per_shard_ssd 8 * osd advanced osd_scrub_auto_repair true mds advanced client_oc false mds advanced client_readahead_max_bytes 4096 mds advanced client_readahead_max_periods 1 mds advanced client_readahead_min 0 mds basic mds_cache_memory_limit 21474836480 client advanced client_oc false client advanced client_readahead_max_bytes 4096 client advanced client_readahead_max_periods 1 client advanced client_readahead_min 0 client advanced fuse_disable_pagecache false The cephfs mount options (note that readahead was disabled for this test): /mnt/cephfs type ceph (rw,relatime,name=cephfs,secret=<hidden>,acl,rasize=0) Any help or pointers are appreciated; this is a major performance issue for us. Thanks and Regards, Ashu Pachauri

4 12

Upgrading to 16.2.11 timing out on ceph-volume due to raw list performance bug, downgrade isn't possible due to new OP code in bluestore
by Mikael Öhman 05 Apr '23

05 Apr '23

Trying to upgrade a containerized setup from 16.2.10 to 16.2.11 gave us two big surprises, I wanted to share in case anyone else encounters the same. I don't see any nice solution to this apart from a new release that fixes the performance regression that completely breaks the container setup in cephadm due to timeouts: After some digging, we would that the it was the "ceph-volume" command that kept timing out, and after a ton of digging, found that it does so because of https://github.com/ceph/ceph/commit/bea9f4b643ce32268ad79c0fc257b25ff2f8333… which was introduced into 16.2.11. Unfortunately, the vital fix for this https://github.com/ceph/ceph/commit/8d7423c3e75afbe111c91e699ef3cb1c0beee61b was not included in 16.2.11 So, in a setup like ours, with *many* devices, a simple "ceph-volume raw list" now takes over 10 minutes to run (instead of 5 seconds in 16.2.10). As a result, the service files that cephadm generates [Service] LimitNOFILE=1048576 LimitNPROC=1048576 EnvironmentFile=-/etc/environment ExecStart=/bin/bash /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/%i/unit.run ExecStop=-/bin/bash -c '/bin/podman stop ceph-5406fed0-d52b-11ec-beff-7ed30a54847b-%i ; bash /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/%i/unit.stop' ExecStopPost=-/bin/bash /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/%i/unit.poststop KillMode=none Restart=on-failure RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=120 StartLimitInterval=30min StartLimitBurst=5 ExecStartPre=-/bin/rm -f %t/%n-pid %t/%n-cid ExecStopPost=-/bin/rm -f %t/%n-pid %t/%n-cid Type=forking PIDFile=%t/%n-pid Delegate=yes will repeatedly be marked as failed, as they take over 2 minutes to run now. This tells systemd to restart, and we now have an infinite loop, as the 5 restarts takes over 50 minutes, it never even triggers the StarLimitInterval, leaving this OSD in an infinite loop over listing the n^2 devices (which, as a bonus, is also filling up the root disk with an enormous amount of repeated logging in ceph-volume.log as it infinitely tries to figure out if a block device is a bluestore) And trying to just fix the service or unit files manually to at least just stop this container from being incorrectly restarted over and over, is also a dead end, since the orchestration stuff just overwrites this automatically, and restarts the services again. I found it seemed to be /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 on my system that generated these files, so i tried tweaking that to have the necessary 1200 second TimeoutStart and finally that managed to get the darn container to stop restarting endlessly. (I admit i'm very fuzzy on how these services and orchestration stuff is triggered as i usually don't work on our storage stuff) Still though, it takes 11 minutes to start each OSD service now, so this isn't great. We wanted to revert back to 16.2.10 but it turns out to also be a no-go, as a new operation added to bluefs https://github.com/ceph/ceph/pull/42750 in 16.2.11 (though this isn't mentioned in the changelogs, i had to compare the source code to see that it was in fact added 16.2.11). So trying to revert an OSD then fails with: debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluefs _replay 0x100000: stop: unrecognized op 12 debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluefs mount failed to replay log: (5) Input/output error debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluestore(/var/lib/ceph/osd/ceph-10) _open_bluefs failed bluefs mount: (5) Input/output error debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluestore(/var/lib/ceph/osd/ceph-10) _open_db failed to prepare db environment: debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 1 bdev(0x5590e80a0400 /var/lib/ceph/osd/ceph-10/block) close debug 2023-04-04T11:42:46.153+0000 7f2c12f6a200 -1 osd.10 0 OSD:init: unable to mount object store debug 2023-04-04T11:42:46.153+0000 7f2c12f6a200 -1 ** ERROR: osd init failed: (5) Input/output error Ouch Best regards, Mikael

3 3

RGW can't create bucket
by Kamil Madac 05 Apr '23

05 Apr '23

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source -- Kamil Madac

3 6

Crushmap rule for multi-datacenter erasure coding
by Michel Jouvin 05 Apr '23

05 Apr '23

Hi, We have a 3-site Ceph cluster and would like to create a 4+2 EC pool with 2 chunks per datacenter, to maximise the resilience in case of 1 datacenter being down. I have not found a way to create an EC profile with this 2-level allocation strategy. I created an EC profile with a failure domain = datacenter but it doesn't work as, I guess, it would like to ensure it has always 5 OSDs up (to ensure that the pools remains R/W) where with a failure domain = datacenter, the guarantee is only 4. My idea was to create a 2-step allocation and a failure domain=host to achieve our desired configuration, with something like the following in the crushmap rule: step choose indep 3 datacenter step chooseleaf indep x host step emit Is it the right approach? If yes, what should be 'x'? Would 0 work? From what I have seen, there is no way to create such a rule with the 'ceph osd crush' commands: I have to download the current CRUSHMAP, edit it and upload the modified version. Am I right? Thanks in advance for your help or suggestions. Best regards, Michel

2 1

Crushmap rule for multi-datacenter erasure coding
by Michel Jouvin 04 Apr '23

04 Apr '23

Hi, We have a 3-site Ceph cluster and would like to create a 4+2 EC pool with 2 chunks per datacenter, to maximise the resilience in case of 1 datacenter being down. I have not found a way to create an EC profile with this 2-level allocation strategy. I created an EC profile with a failure domain = datacenter but it doesn't work as, I guess, it would like to ensure it has always 5 OSDs up (to ensure that the pools remains R/W) where with a failure domain = datacenter, the guarantee is only 4. My idea was to create a 2-step allocation and a failure domain=host to achieve our desired configuration, with something like the following in the crushmap rule: step choose indep 3 datacenter step chooseleaf indep x host step emit Is it the right approach? If yes, what should be 'x'? Would 0 work? From what I have seen, there is no way to create such a rule with the 'ceph osd crush' commands: I have to download the current CRUSHMAP, edit it and upload the modified version. Am I right? Thanks in advance for your help or suggestions. Best regards, Michel

3 5

Recently deployed cluster showing 9Tb of raw usage without any load deployed
by Work Ceph 04 Apr '23

04 Apr '23

Hello guys! We noticed an unexpected situation. In a recently deployed Ceph cluster we are seeing a raw usage, that is a bit odd. We have the following setup: We have a new cluster with 5 nodes with the following setup: - 128 GB of RAM - 2 cpus Intel(R) Intel Xeon Silver 4210R - 1 NVME of 2 TB for the rocks DB caching - 5 HDDs of 14TB - 1 NIC dual port of 25GiB in BOND mode. Right after deploying the Ceph cluster, we see a raw usage of about 9TiB. However, no load has been applied onto the cluster. Have you guys seen such a situation? Or, can you guys help understand it? We are using Ceph Octopus, and we have set the following configurations: ``` ceph_conf_overrides: global: osd pool default size: 3 osd pool default min size: 1 osd pool default pg autoscale mode: "warn" perf: true rocksdb perf: true mon: mon osd down out interval: 120 osd: bluestore min alloc size hdd: 65536 ``` Any tip or help on how to explain this situation is welcome!

3 7

Read and write performance on distributed filesystem
by David Cunningham 04 Apr '23

04 Apr '23

Hello, We are considering CephFS as an alternative to GlusterFS, and have some questions about performance. Is anyone able to advise us please? This would be for file systems between 100GB and 2TB in size, average file size around 5MB, and a mixture of reads and writes. I may not be using the correct terminology in the Ceph world, but in my parlance a node is a Linux server running the Ceph storage software. Multiple nodes make up the whole Ceph storage solution. Someone correct me if I should be using different terms! In our normal scenario the nodes in the replicated filesystem would be around 0.3ms apart, but we're also interested in geographically remote nodes which would be say 20ms away. We are using third party software which relies on a traditional Linux filesystem, so we can't use an object storage solution directly. So my specific questions are: 1. When reading a file from CephFS, does it read from just one node, or from all nodes? 2. If reads are from one node then does it choose the node with the fastest response to optimise performance, or if from all nodes then will reads be no faster than latency to the furthest node? 3. When writing to CephFS, are all nodes written to synchronously, or are writes to one node which then replicates that to other nodes asynchronously? 4. Can anyone give a recommendation on maximum latency between nodes to have decent performance? 5. How does CephFS handle a node which suddenly becomes unavailable on the network? Is the block time configurable, and how good is the healing process after the lost node rejoins the network? 6. I have read that CephFS is more complicated to administer than GlusterFS. What does everyone think? Are things like healing after a net split difficult for administrators new to Ceph to handle? Thanks very much in advance. -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782

2 1

compiling Nautilus for el9
by Marc 03 Apr '23

03 Apr '23

Is it possible to compile Nautilus for el9? Or maybe just the osd's?

2 4

How mClock profile calculation works, and IOPS
by Luis Domingues 03 Apr '23

03 Apr '23

Hi, I am reading reading some documentation about mClock and have two questions. First, about the IOPS. Are those IOPS disk IOPS or other kind of IOPS? And what the assumption of those? (Like block size, sequential or random reads/writes)? And the second question, How mClock calculates its profiles? I have my lab cluster running Quincy, and I have this parameters for mClock: "osd_mclock_max_capacity_iops_hdd": "450.000000", "osd_mclock_profile": "balanced", According to the documentation: https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#bala… I am expecting to have: "osd_mclock_scheduler_background_best_effort_lim": "999999", "osd_mclock_scheduler_background_best_effort_res": "90", "osd_mclock_scheduler_background_best_effort_wgt": "2", "osd_mclock_scheduler_background_recovery_lim": "675", "osd_mclock_scheduler_background_recovery_res": "180", "osd_mclock_scheduler_background_recovery_wgt": "1", "osd_mclock_scheduler_client_lim": "450", "osd_mclock_scheduler_client_res": "180", "osd_mclock_scheduler_client_wgt": "1", But what I get is: "osd_mclock_scheduler_background_best_effort_lim": "999999", "osd_mclock_scheduler_background_best_effort_res": "18", "osd_mclock_scheduler_background_best_effort_wgt": "2", "osd_mclock_scheduler_background_recovery_lim": "135", "osd_mclock_scheduler_background_recovery_res": "36", "osd_mclock_scheduler_background_recovery_wgt": "1", "osd_mclock_scheduler_client_lim": "90", "osd_mclock_scheduler_client_res": "36", "osd_mclock_scheduler_client_wgt": "1", Which seems very low according to what my disk seems to be able to handle. Is this calculation the expected one? Or did I miss something on how those profiles are populated? Luis Domingues Proton AG

2 5

Eccessive occupation of small OSDs
by Nicola Mori 03 Apr '23

03 Apr '23

Dear Ceph users, my cluster is made up of 10 old machines, with uneven number of disks and disk size. Essentially I have just one big data pool (6+2 erasure code, with host failure domain) for which I am currently experiencing a very poor available space (88 TB of which 40 TB occupied, as reported by df -h on hosts mounting the cephfs) compared to the raw one (196.5 TB). I have a total of 104 OSDs and 512 PGs for the pool; I cannot increment the PG number since the machines are old and with very low amount of RAM, and some of them are already overloaded. In this situation I'm seeing a high occupation of small OSDs (500 MB) with respect to bigger ones (2 and 4 TB) even if the weight is set equal to disk capacity (see below for ceph osd tree). For example OSD 9 is at 62% occupancy even with weight 0.5 and reweight 0.75, while the highest occupancy for 2 TB OSDs is 41% (OSD 18) and 4 TB OSDs is 23% (OSD 79). I guess this high occupancy for 500 MB OSDs combined with erasure code size and host failure domain might be the cause of the poor available space, could this be true? The upmap balancer is currently running but I don't know if and how much it could improve the situation. Any hint is greatly appreciated, thanks. Nicola # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 196.47754 root default -7 14.55518 host aka 4 hdd 1.81940 osd.4 up 1.00000 1.00000 11 hdd 1.81940 osd.11 up 1.00000 1.00000 18 hdd 1.81940 osd.18 up 1.00000 1.00000 26 hdd 1.81940 osd.26 up 1.00000 1.00000 32 hdd 1.81940 osd.32 up 1.00000 1.00000 41 hdd 1.81940 osd.41 up 1.00000 1.00000 48 hdd 1.81940 osd.48 up 1.00000 1.00000 55 hdd 1.81940 osd.55 up 1.00000 1.00000 -3 14.55518 host balin 0 hdd 1.81940 osd.0 up 1.00000 1.00000 8 hdd 1.81940 osd.8 up 1.00000 1.00000 15 hdd 1.81940 osd.15 up 1.00000 1.00000 22 hdd 1.81940 osd.22 up 1.00000 1.00000 29 hdd 1.81940 osd.29 up 1.00000 1.00000 34 hdd 1.81940 osd.34 up 1.00000 1.00000 43 hdd 1.81940 osd.43 up 1.00000 1.00000 49 hdd 1.81940 osd.49 up 1.00000 1.00000 -13 29.10950 host bifur 3 hdd 3.63869 osd.3 up 1.00000 1.00000 14 hdd 3.63869 osd.14 up 1.00000 1.00000 27 hdd 3.63869 osd.27 up 1.00000 1.00000 37 hdd 3.63869 osd.37 up 1.00000 1.00000 50 hdd 3.63869 osd.50 up 1.00000 1.00000 59 hdd 3.63869 osd.59 up 1.00000 1.00000 64 hdd 3.63869 osd.64 up 1.00000 1.00000 69 hdd 3.63869 osd.69 up 1.00000 1.00000 -17 29.10950 host bofur 2 hdd 3.63869 osd.2 up 1.00000 1.00000 21 hdd 3.63869 osd.21 up 1.00000 1.00000 39 hdd 3.63869 osd.39 up 1.00000 1.00000 57 hdd 3.63869 osd.57 up 1.00000 1.00000 66 hdd 3.63869 osd.66 up 1.00000 1.00000 72 hdd 3.63869 osd.72 up 1.00000 1.00000 76 hdd 3.63869 osd.76 up 1.00000 1.00000 79 hdd 3.63869 osd.79 up 1.00000 1.00000 -21 29.10376 host dwalin 88 hdd 1.81898 osd.88 up 1.00000 1.00000 89 hdd 1.81898 osd.89 up 1.00000 1.00000 90 hdd 1.81898 osd.90 up 1.00000 1.00000 91 hdd 1.81898 osd.91 up 1.00000 1.00000 92 hdd 1.81898 osd.92 up 1.00000 1.00000 93 hdd 1.81898 osd.93 up 1.00000 1.00000 94 hdd 1.81898 osd.94 up 1.00000 1.00000 95 hdd 1.81898 osd.95 up 1.00000 1.00000 96 hdd 1.81898 osd.96 up 1.00000 1.00000 97 hdd 1.81898 osd.97 up 1.00000 1.00000 98 hdd 1.81898 osd.98 up 1.00000 1.00000 99 hdd 1.81898 osd.99 up 1.00000 1.00000 100 hdd 1.81898 osd.100 up 1.00000 1.00000 101 hdd 1.81898 osd.101 up 1.00000 1.00000 102 hdd 1.81898 osd.102 up 1.00000 1.00000 103 hdd 1.81898 osd.103 up 1.00000 1.00000 -9 14.55518 host ogion 7 hdd 1.81940 osd.7 up 1.00000 1.00000 16 hdd 1.81940 osd.16 up 1.00000 1.00000 23 hdd 1.81940 osd.23 up 1.00000 1.00000 33 hdd 1.81940 osd.33 up 1.00000 1.00000 40 hdd 1.81940 osd.40 up 1.00000 1.00000 47 hdd 1.81940 osd.47 up 1.00000 1.00000 54 hdd 1.81940 osd.54 up 1.00000 1.00000 61 hdd 1.81940 osd.61 up 1.00000 1.00000 -19 14.55518 host prestno 81 hdd 1.81940 osd.81 up 1.00000 1.00000 82 hdd 1.81940 osd.82 up 1.00000 1.00000 83 hdd 1.81940 osd.83 up 1.00000 1.00000 84 hdd 1.81940 osd.84 up 1.00000 1.00000 85 hdd 1.81940 osd.85 up 1.00000 1.00000 86 hdd 1.81940 osd.86 up 1.00000 1.00000 87 hdd 1.81940 osd.87 up 1.00000 1.00000 104 hdd 1.81940 osd.104 up 1.00000 1.00000 -15 29.10376 host remolo 6 hdd 1.81897 osd.6 up 1.00000 1.00000 12 hdd 1.81897 osd.12 up 1.00000 1.00000 19 hdd 1.81897 osd.19 up 1.00000 1.00000 28 hdd 1.81897 osd.28 up 1.00000 1.00000 35 hdd 1.81897 osd.35 up 1.00000 1.00000 44 hdd 1.81897 osd.44 up 1.00000 1.00000 52 hdd 1.81897 osd.52 up 1.00000 1.00000 58 hdd 1.81897 osd.58 up 1.00000 1.00000 63 hdd 1.81897 osd.63 up 1.00000 1.00000 67 hdd 1.81897 osd.67 up 1.00000 1.00000 71 hdd 1.81897 osd.71 up 1.00000 1.00000 73 hdd 1.81897 osd.73 up 1.00000 1.00000 74 hdd 1.81897 osd.74 up 1.00000 1.00000 75 hdd 1.81897 osd.75 up 1.00000 1.00000 77 hdd 1.81897 osd.77 up 1.00000 1.00000 78 hdd 1.81897 osd.78 up 1.00000 1.00000 -5 14.55518 host rokanan 1 hdd 1.81940 osd.1 up 1.00000 1.00000 10 hdd 1.81940 osd.10 up 1.00000 1.00000 17 hdd 1.81940 osd.17 up 1.00000 1.00000 24 hdd 1.81940 osd.24 up 1.00000 1.00000 31 hdd 1.81940 osd.31 up 1.00000 1.00000 38 hdd 1.81940 osd.38 up 1.00000 1.00000 46 hdd 1.81940 osd.46 up 1.00000 1.00000 53 hdd 1.81940 osd.53 up 1.00000 1.00000 -11 7.27515 host romolo 5 hdd 0.45470 osd.5 up 1.00000 1.00000 9 hdd 0.45470 osd.9 up 0.75000 1.00000 13 hdd 0.45470 osd.13 up 1.00000 1.00000 20 hdd 0.45470 osd.20 up 0.95000 1.00000 25 hdd 0.45470 osd.25 up 0.75000 1.00000 30 hdd 0.45470 osd.30 up 1.00000 1.00000 36 hdd 0.45470 osd.36 up 1.00000 1.00000 42 hdd 0.45470 osd.42 up 1.00000 1.00000 45 hdd 0.45470 osd.45 up 0.85004 1.00000 51 hdd 0.45470 osd.51 up 0.89999 1.00000 56 hdd 0.45470 osd.56 up 1.00000 1.00000 60 hdd 0.45470 osd.60 up 1.00000 1.00000 62 hdd 0.45470 osd.62 up 1.00000 1.00000 65 hdd 0.45470 osd.65 up 0.85004 1.00000 68 hdd 0.45470 osd.68 up 1.00000 1.00000 70 hdd 0.45470 osd.70 up 1.00000 1.00000

4 5

Re: monitoring apply_latency / commit_latency ?
by Konstantin Shalygin 03 Apr '23

03 Apr '23

Hi, > On 2 Apr 2023, at 23:14, Matthias Ferdinand <mf+ml.ceph(a)mfedv.net> wrote: > > I understand that grafana graphs are generated from prometheus metrics. > I just wanted to know which OSD daemon-perf values feed these prometheus > metrics (or if they are generated in some other way). Yep, this perf metrics is generated in some way 🙂 You can consult with ceph-mgr prometheus module source code [1] [1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/prometheus/module.py#… k

1 0

Set the Quality of Service configuration.
by farhad kh 02 Apr '23

02 Apr '23

how to i can set IO quota or limit for R/W for erasure coding pool in ceph ?

3 2

monitoring apply_latency / commit_latency ?
by Matthias Ferdinand 02 Apr '23

02 Apr '23

Hi, I would like to understand how the per-OSD data from "ceph osd perf" (i.e. apply_latency, commit_latency) is generated. So far I couldn't find documentation on this. "ceph osd perf" output is nice for a quick glimpse, but is not very well suited for graphing. Output values are from the most recent 5s-averages apparently. With "ceph daemon osd.X perf dump" OTOH, you get quite a lot of latency metrics, while it is just not obvious to me how they aggregate into apply_latency and commit_latency. Or some comparably easy read latency metric (something that is missing completely in "ceph osd perf"). Can somebody shed some light on this? Regards Matthias

2 4

avg apply latency went up after update from octopus to pacific
by Boris Behrens 02 Apr '23

02 Apr '23

Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms. All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value? Cheers Boris

10 27

Failing to create monitor in a working cluster.
by pmestre＠gmail.com 01 Apr '23

01 Apr '23

Hello, i've been running a 3 node proxmox cluster with 4 ceph osd for 3 years as a production cluster. As a test for trying to move ceph cluster network, i destroyed one of the 3 working monitors and tried to recreate it. After destroying it, the new monitor refuses to join the cluster, even in the old network. I've tried all steps in documentation "Troubleshooting monitors" section. New monitor has this config extracted from ceph --admin-daemon file.asok { "name": "n3ceph", "rank": -1, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "2449958197560098820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "0", "quorum_mon": [] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false } The quorum mon stat is as follows: { "name": "node1", "rank": 0, "state": "leader", "election_epoch": 340, "quorum": [ 0, 1 ], "quorum_age": 13090, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138314316775423", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 5 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 1 }, { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 12 } ], "mgr": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false I tried to get a debug log with ceph daemon mon.n3ceph config set debug_mon 10/10 and restarting the service, but the ceph log file stoped working after i tried that setting. journalctl -u tells me: mar 31 17:35:22 node3 ceph-mon[240916]: 2023-03-31T17:35:22.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:27 node3 ceph-mon[240916]: 2023-03-31T17:35:27.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:32 node3 ceph-mon[240916]: 2023-03-31T17:35:32.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200). Any ideas? Cluster is running fine with two monitors, but a reboot in one of the nodes might be a big problem. Kind regards and many thanks.

2 1