April 2023 - ceph-users - lists.ceph.io

MGR Memory Leak in Restful
by Chris Palmer 17 Apr '23

17 Apr '23

We've hit a memory leak in the Manager Restful interface, in versions 17.2.5 & 17.2.6. On our main production cluster the active MGR grew to about 60G until the oom_reaper killed it, causing a successful failover and restart of the failed one. We can then see that the problem is recurring, actually on all 3 of our clusters. We've traced this to when we enabled full Ceph monitoring by Zabbix last week. The leak is about 20GB per day, and seems to be proportional to the number of PGs. For some time we just had the default settings, and no memory leak, but had not got around to finding why many of the Zabbix items were showing as Access Denied. We traced this to the MGR's MON CAPS which were "mon 'profile mgr'". The MON logs showed recurring: log_channel(audit) log [DBG] : from='mgr.284576436 192.168.xxx.xxx:0/2356365' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: access denied Changing the MGR CAPS to "mon 'allow *'" and restarting the MGR immediately allowed that to work, and all the follow-on REST calls worked. log_channel(audit) log [DBG] : from='mgr.283590200 192.168.xxx.xxx:0/1779' entity='mgr.host1' cmd=[{"format": "json", "prefix": "pg dump"}]: dispatch However it has also caused the memory leak to start. We've reverted the CAPS and are back to how we were. Two questions: 1) No matter what the REST consumer is doing, the MGR should not accumulate memory, especially as we can see that the REST TCP connections have wrapped up. Is there anything more we can do to diagnose this? 2) Setting "allow *" worked, but is there are better setting just to allow the "pg dump" call (in addition to profile mgr)? Thanks, Chris

1 0

unable to deploy ceph -- failed to read label for XXX No such file or directory
by Radoslav Bodó 17 Apr '23

17 Apr '23

hello, during basic experimentation I'm running into wierd situaltion when adding osd to test cluster. The test cluster is created as 3x XEN DomU Debian Bookworm (test1-3), 4x CPU, 8GB RAM, xvda root, xvbd swap, 4x xvdj,k,l,m 20GB (LVM volumes in Dom0, propagated via xen phy device) and cleaned with `wipefs -a` ``` apt-get install cephadm ceph-common cephadm bootstrap --mon-ip 10.0.0.101 ceph orch host add test2 ceph orch host add test3 ``` when adding OSDs the first host gets created OSDs as expected, but during creating OSDs on second host the output gets wierd, even when adding each device separately the output shows that `ceph orch` tries to create multiple osds at once ``` root@test1:~# for xxx in j k l m; do ceph orch daemon add osd test2:/dev/xvd$xxx; done Created osd(s) 0,1,2,3 on host 'test2' Created osd(s) 0,1 on host 'test2' Created osd(s) 2,3 on host 'test2' Created osd(s) 1 on host 'test2' ``` the syslog on test2 node shows an errors ``` 2023-04-16T20:57:02.528456+00:00 test2 bash[10426]: cephadm 2023-04-16T20:57:01.389951+0000 mgr.test1.ucudzp (mgr.14206) 1691 : cephadm [INF] Found duplicate OSDs: osd.0 in status running on test1, osd.0 in status error on test2 2023-04-16T20:57:02.528748+00:00 test2 bash[10426]: cephadm 2023-04-16T20:57:01.391346+0000 mgr.test1.ucudzp (mgr.14206) 1692 : cephadm [INF] Removing daemon osd.0 from test2 -- ports [] 2023-04-16T20:57:02.528943+00:00 test2 bash[10426]: cluster 2023-04-16T20:57:02.350564+0000 mon.test1 (mon.0) 743 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2023-04-16T20:57:17.972962+00:00 test2 bash[20098]: stderr: failed to read label for /dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d: (2) No such file or directory 2023-04-16T20:57:17.973064+00:00 test2 bash[20098]: stderr: 2023-04-16T20:57:17.962+0000 7fad2451c540 -1 bluestore(/dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d) _read_bdev_label failed to open /dev/ceph-48f3646c-7070-4a37-b9a4-ed0a4a983965/osd-block-11a0dc2b-f8e1-4694-813f-2309ab6a5c1d: (2) No such file or directory 2023-04-16T20:57:17.973181+00:00 test2 bash[20098]: --> Failed to activate via lvm: command returned non-zero exit status: 1 2023-04-16T20:57:17.973278+00:00 test2 bash[20098]: --> Failed to activate via simple: 'Namespace' object has no attribute 'json_config' 2023-04-16T20:57:17.973368+00:00 test2 bash[20098]: --> Failed to activate any OSD(s) ``` the ceph and cephadm binaries are installed from debian bookworm ``` ii ceph-common 16.2.11+ds-2 amd64 common utilities to mount and interact with a ceph storage cluster ii cephadm 16.2.11+ds-2 amd64 utility to bootstrap ceph daemons with systemd and containers ``` management session script can be found at https://pastebin.com/raw/FiX7DMHS none of the googled symptoms helped me to understand why is this situation happening nor how to troubleshoot or debug the issues. I'd understand that the nodes are very log on RAM to get this experiment running, but the behavior does not really look like OOM issue. any idea would be appreciated thanks bodik

1 1

RADOSGW zone data-pool migration.
by Gaël THEROND 17 Apr '23

17 Apr '23

Hi everyone, quick question regarding radosgw zone data-pool. I’m currently planning to migrate an old data-pool that was created with inappropriate failure-domain to a newly created pool with appropriate failure-domain. If I’m doing something like: radosgw-admin zone modify —rgw-zone default —data-pool <new_pool> Will data from the old pool be migrated to the new one or do I need to do something else to migrate those data out of the old pool? I’ve read a lot of mail archive with peoples willing to do that but I can’t get a clear answer from those archives. I’m running on nautilus release of it ever help. Thanks a lot!

1 0

Radosgw-admin bucket list has duplicate objects
by mahnoosh shahidi 17 Apr '23

17 Apr '23

Hi, I observed duplicate object names in the result of the admin list bucket on 15.2.12 cluster. I used the following command and some of the object names in the result list appeared more than once. There is no versioning config for the bucket. radosgw-admin bucket list --allow-unordered --max-entries=200000000 > --bucket=my-bucket > my-bucket.list Does anyone have any opinion on this? Should I make a bug report for this? Best Regards, Mahnoosh

3 3

OSDs remain not in after update to v17
by Alexandre Becholey 17 Apr '23

17 Apr '23

Dear Ceph Users, I have a small ceph cluster for VMs on my local machine. It used to be installed with the system packages and I migrated it to docker following the documentation. It worked OK until I migrated from v16 to v17 a few months ago. Now the OSDs remain "not in" as shown in the status: # ceph -s cluster: id: abef2e91-cd07-4359-b457-f0f8dc753dfa health: HEALTH_WARN 6 stray daemon(s) not managed by cephadm 1 stray host(s) with 6 daemon(s) not managed by cephadm 2 devices (4 osds) down 4 osds down 1 host (4 osds) down 1 root (4 osds) down Reduced data availability: 129 pgs inactive services: mon: 1 daemons, quorum bjorn (age 8m) mgr: bjorn(active, since 8m) osd: 4 osds: 0 up (since 4w), 4 in (since 4w) data: pools: 2 pools, 129 pgs objects: 0 objects, 0 B usage: 1.8 TiB used, 1.8 TiB / 3.6 TiB avail pgs: 100.000% pgs unknown 129 unknown I can see some network communication between the OSDs and the monitor and the OSDs are running: # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f8fbe8177a63 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-2 6768ec871404 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-1 ff82f84504d5 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-0 4c89e50ce974 quay.io/ceph/ceph:v17 "/usr/bin/ceph-osd -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-osd-3 fe0b6089edda quay.io/ceph/ceph:v17 "/usr/bin/ceph-mon -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mon-bjorn f76ac9dcdd6d quay.io/ceph/ceph:v17 "/usr/bin/ceph-mgr -…" 9 minutes ago Up 9 minutes ceph-abef2e91-cd07-4359-b457-f0f8dc753dfa-mgr-bjorn However when I try to use any `ceph orch` commands, they hang. I can also see some blacklist on the OSDs: # ceph osd blocklist ls 10.99.0.13:6833/3770763474 2023-04-13T08:17:38.885128+0000 10.99.0.13:6832/3770763474 2023-04-13T08:17:38.885128+0000 10.99.0.13:0/2634718754 2023-04-13T08:17:38.885128+0000 10.99.0.13:0/1103315748 2023-04-13T08:17:38.885128+0000 listed 4 entries The first two entries correspond to the manager process. `ceph osd blocked-by` does not show anything. I think I might have forgotten to set the `ceph osd require-osd-release ...` because 14 is written in `/var/lib/ceph/<ID>/osd.?/require_osd_release`. If I try to do it now, the monitor hits an abort: debug 0> 2023-04-12T08:43:27.788+0000 7f0fcf2aa700 -1 *** Caught signal (Aborted) ** in thread 7f0fcf2aa700 thread_name:ms_dispatch ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f0fd94bbcf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f0fdb5124e3] 5: /usr/lib64/ceph/libceph-common.so.2(+0x26a64f) [0x7f0fdb51264f] 6: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basi 7: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x38d) [0x562719cb127d] 8: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x17b) [0x562719cb18cb] 9: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x2ce) [0x562719c20ade] 10: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x1ebb) [0x562719ab9f6b] 11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x9f2) [0x562719abe152] 12: (Monitor::_ms_dispatch(Message*)+0x406) [0x562719abf066] 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x562719aef13d] 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7f0fdb78e0e8] 15: (DispatchQueue::entry()+0x50f) [0x7f0fdb78b52f] 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0fdb8543b1] 17: /lib64/libpthread.so.0(+0x81ca) [0x7f0fd94b11ca] 18: clone() Any ideas on what is going on? Many thanks, Alexandre

3 4

CEPH Mirrors are lacking packages
by Burkhard Linke 17 Apr '23

17 Apr '23

Hi, at least eu.ceph.com and de.ceph.com are lacking packages for the pacific release. All package not start with "c" (e.g. librbd, librados, radosgw) are missing. Best regards, Burkhard Linke

3 2

Cannot add disks back after their OSDs were drained and removed from a cluster
by stillsmil＠gmail.com 17 Apr '23

17 Apr '23

I find that I cannot re-add a disk to a Ceph cluster after the OSD on the disk is removed. Ceph seems to know about the existence of these disks, but not about their "host:dev" information: ``` # ceph device ls DEVICE HOST:DEV DAEMONS WEAR LIFE EXPECTANCY SAMSUNG_MZ7L37T6_S6KHNE0T100049 0% <-- should be host01:sda, was osd.0 SAMSUNG_MZ7L37T6_S6KHNE0T100050 host01:sdb osd.1 0% SAMSUNG_MZ7L37T6_S6KHNE0T100052 0% <-- should be host02:sda SAMSUNG_MZ7L37T6_S6KHNE0T100053 host01:sde osd.9 0% SAMSUNG_MZ7L37T6_S6KHNE0T100061 host01:sdf osd.11 0% SAMSUNG_MZ7L37T6_S6KHNE0T100062 0% SAMSUNG_MZ7L37T6_S6KHNE0T100063 host01:sdc osd.5 0% SAMSUNG_MZ7L37T6_S6KHNE0T100064 host01:sdg osd.13 0% SAMSUNG_MZ7L37T6_S6KHNE0T100065 0% SAMSUNG_MZ7L37T6_S6KHNE0T100066 host01:sdd osd.7 0% SAMSUNG_MZ7L37T6_S6KHNE0T100067 0% SAMSUNG_MZ7L37T6_S6KHNE0T100068 0% <-- should be host02:sdb SAMSUNG_MZ7L37T6_S6KHNE0T100069 0% SAMSUNG_MZ7L37T6_S6KHNE0T100070 0% SAMSUNG_MZ7L37T6_S6KHNE0T100071 0% SAMSUNG_MZ7L37T6_S6KHNE0T100072 host01:sdh osd.15 0% SAMSUNG_MZQL27T6HBLA-00B7C_S6CVNG0T321600 host03:nvme4n1 osd.20 0% ... obmitted ... SAMSUNG_MZQL27T6HBLA-00B7C_S6CVNG0T321608 host03:nvme8n1 osd.22 0% ``` For disk "SAMSUNG_MZ7L37T6_S6KHNE0T100049", the "HOST:DEV" field is empty, while I believe it should be "host01:sda", as I have confirmed by running `smartctl -i /dev/sda" on host01. I guess the missing information is the reason that OSDs cannot be created, either manually or automatically on these devices. I have tried: 1. `ceph orch daemon add osd host01:/dev/sda` Prints "Created no osd(s) on host host01; already created?" 2. `ceph orch apply osd --all-available-devices` adds nothing I arrived at such situation while testing if draining a host works: I drained host02, removed it and added it back via: ``` ceph orch host drain host02 ceph orch host rm host02 ceph orch host add host02 <internal_ip> --labels _admin ``` I am running Ceph 17.2.6 on ubuntu. Ceph was deployed via cephadm. FYI the relavent orchestrator spec for osd: ``` # ceph orch ls osd --export service_type: osd service_name: osd unmanaged: true spec: filter_logic: AND objectstore: bluestore --- service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore ``` Any thoughts on what maybe wrong here? Is there a way that I can tell Ceph "you are wrong about the whereabouts of these disks, forget what you know and fetch disk information afresh"? Any help much appreciated!

1 0

Ceph mon status stuck at "probing"
by York Huang 16 Apr '23

16 Apr '23

Hi Ceph community, I had a small lab cluster of version Nautilus (ceph-ansible / containerized) which functions quite well. As part of the upgrade experiment, I replaced one of the mon with an Octopus one (containerized as well, the OS is purged before mon deployment), the daemon seems working but its status is always "probing", and "ceph -s" showed that mon is down and out of quorum. I checked and compared the output of "ceph daemon XXX mon_status" of all monitors, ensuring there is no network issue, the new mon seemed to have the correct monmap but always stucked in status "probing". I doubt if Ceph mon of different versions do co-exist. Any further steps suggest? Thanks in advanced.

1 0

Dead node (watcher) won't timeout on RBD
by Max Boone 16 Apr '23

16 Apr '23

After a critical node failure on my lab cluster, which won't come back up and is still down, the RBD objects are still being watched / mounted according to ceph. I can't shell to the node to rbd unbind them as the node is down. I am absolutely certain that nothing is using these images and they don't have snapshots either (and this IP is not even remotely close to the those of the monitors in the cluster). I blocked the IP usingceph osd blocklist add but after 30 minutes, they are still being watched. Them being watched (they are RWO ceph-csi volumes) prevents me from re-using them in the cluster. As far as I'm aware, ceph should remove the watchers after 30 minutes and they've been blocklisted for hours now.root@node0:~# rbd status kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff Watchers: watcher=10.0.0.103:0/992994811 client.1634081 cookie=139772597209280 root@node0:~# rbd snap list kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff root@node0:~# rbd info kubernetes/csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff rbd image 'csi-vol-e6a07ccd-93f6-4c47-a948-201501440fff': size 10 GiB in 2560 objects order 22 (4 MiB objects) snapshot_count: 0 id: 4ff5353b865e1 block_name_prefix: rbd_data.4ff5353b865e1 format: 2 features: layering op_features: flags: create_timestamp: Fri Mar 31 14:46:51 2023 access_timestamp: Fri Mar 31 14:46:51 2023 modify_timestamp: Fri Mar 31 14:46:51 2023 root@node0:~# rados -p kubernetes listwatchers rbd_header.4ff5353b865e1 watcher=10.0.0.103:0/992994811 client.1634081 cookie=139772597209280 root@node0:~# ceph osd blocklist ls 10.0.0.103:0/0 2023-04-16T13:58:34.854232+0200 listed 1 entries root@node0:~# ceph daemon osd.0 config get osd_client_watch_timeout { "osd_client_watch_timeout": "30" } Is it possible to kick a watcher out manually, or is there not much I can do here besides shutting down the entire cluster (or OSDs) and getting them back up? If it is a bug, I'm happy to help figuring out it's root cause and see if I can help writing a fix. Cheers, Max.

2 1

Re: Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 15 Apr '23

15 Apr '23

It's funny, I had the EPEL repos added yesterday, and no joy. Left it alone for a day, and voila, packages exist. Must have forgotten to update or clear my cache or something? Thanks for the help!

1 0

Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 15 Apr '23

15 Apr '23

Hello! I'm trying to Install the ceph-common package on a Rocky Linux 9 box so that I can connect to our ceph cluster and mount user directories. I've added the ceph repo to yum.repos.d, but when I run `dnf install ceph-common`, I get the following error ``` [root@jet yum.repos.d]# dnf install ceph-common Last metadata expiration check: 0:33:06 ago on Fri 14 Apr 2023 11:29:26 AM EDT. Error: Problem: conflicting requests - nothing provides libthrift-0.14.0.so()(64bit) needed by ceph-common-2:17.2.6-0.el9.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) ``` I've looked far and wide for this libthrift package, but I can't figure out what repo to add to get it. Has anyone had success installing ceph on Rocky 9 and would be willing to share some guidance?

2 1

ceph/daemon stable tag
by Jonas Nemeikšis 15 Apr '23

15 Apr '23

Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas

2 6

Mysteriously dead OSD process
by J-P Methot 15 Apr '23

15 Apr '23

Hi, We currently use Ceph Pacific 16.2.10 deployed with Cephadm on this storage cluster. Last night, one of our OSD died. However, since its storage is a SSD, we ran hardware checks and found no issue with the SSD itself. However, if we try starting the service again, the container just crashes 1 second after booting up. If I look at the logs, there's no error. You can see the OSD starting up normally and then the last line before the crash is : debug 2023-04-05T18:32:57.433+0000 7f8078e0c700 1 osd.87 pg_epoch: 207175 pg[2.99s3( v 207174'218628609 (207134'218623666,207174'218628609] local-lis/les=207140/207141 n=38969 ec=41966/315 lis/c=207140/207049 les/c/f=207141/207050/0 sis=207175 pruub=11.464111328s) [5,228,217,NONE,17,25,167,114,158,178,159]/[5,228,217,87,17,25,167,114,158,178,159]p5(0) r=3 lpr=207175 pi=[207049,207175)/1 crt=207174'218628605 mlcod 0'0 remapped NOTIFY pruub 12054.601562500s@ mbc={}] state<Start>: transitioning to Stray I don't really see how this line could cause the OSD to crash. Systemd just writes : Stopping Ceph osd.83 for (uuid) What could cause this OSD to boot up and then suddenly die? Outside the ceph daemon logs and the systemd logs, is there another way I could gain more information? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

2 1

Re: Nothing provides libthrift-0.14.0.so()(64bit)
by Will Nilges 14 Apr '23

14 Apr '23

Oops, forgot to mention that I'm installing Ceph 17.2.6, preempting an upgrade of our cluster from 15.2.17 to 17.2.6.

1 0

17.2.6 Dashboard/RGW Signature Mismatch
by Chris Palmer 14 Apr '23

14 Apr '23

Hi I have 3 Ceph clusters, all configured similarly, which have been happy for some months on 17.2.5: 1. A test cluster 2. A small production cluster 3. A larger production cluster All are debian 11 built from packages - no cephadm. I upgraded (1) to 17.2.6 without any problems at all. In particular the Object Gateway sections of the dashboard work as usual. I then upgraded (2). Nothing seemed amiss, and everything seems to work except... when I try to access the Object Gateway sections of the dashboard I always get: *The Object Gateway Service is not configured* Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","RequestId":"tx0000022ba920e82ac4a9c-0064381' b'934-10e73385-default","HostId":"10e73385-default-default"}') (Just the RequestId changes each time). Before the upgrade it worked just fine. Other info: * RGW requests using awscli and rclone all work with normal RGW accounts. It just seems to be the dashboard that's died. * Just the one zonegroup, no multisite/replication * "radosgw-admin user info --uid=rgwadmin" gives the correct output with the right access_key & secret_key. The other fields are as in (1). * "ceph dashboard get-rgw-api-access-key/get-rgw-api-secret-key" both give the right values. The rgw logs from (2) which fails show: 2023-04-13T16:36:28.720+0100 7fcc7966a700 1 ====== starting new request req=0x7fcd88c10720 ===== 2023-04-13T16:36:28.720+0100 7fcc80e79700 1 req 8090309398268968541 0.000000000s op->ERRORHANDLER: err_no=-2027 new_err_no=-2027 2023-04-13T16:36:28.724+0100 7fcc80e79700 1 ====== req done req=0x7fcd88c10720 op status=0 http_status=403 latency=0.003999980s ====== 2023-04-13T16:36:28.724+0100 7fcc80e79700 1 beast: 0x7fcd88c10720: 192.168.xx.xx - - [13/Apr/2023:16:36:28.720 +0100] "GET /admin/metadata/user?myself HTTP/1.1" 403 134 - "python-requests/2.25.1" - latency=0.003999980s (Note this does not have rgwadmin as the user, and is always the same URL) Whereas the rgw logs from (1) which works show things like: 2023-04-13T15:44:19.396+0000 7f8478da1700 1 ====== starting new request req=0x7f86284f5720 ===== 2023-04-13T15:44:19.412+0000 7f8478da1700 1 ====== req done req=0x7f86284f5720 op status=0 http_status=200 latency=0.016000060s ====== 2023-04-13T15:44:19.412+0000 7f8478da1700 1 beast: 0x7f86284f5720: 10.xx.xx.xx - rgwadmin [13/Apr/2023:15:44:19.396 +0000] "GET /admin/realm?list HTTP/1.1" 200 31 - "python-requests/2.25.1" - latency=0.016000060s (Note this has rgwadmin as the user, and various URLs) The only thing I can see in the release notes that looks even vaguely related is https://github.com/ceph/ceph/pull/47547, but it doesn't seem likely. I am really stumped on this, with no idea what has gone wrong on (2), and what the difference is between (1) and (2). I'm not going to touch (3) until I have resolved this. Grateful for any help... And thanks for all the good work. Regards, Chris

3 4

Restrict user to an RBD image in a pool
by Work Ceph 14 Apr '23

14 Apr '23

Hello guys! Is it possible to restrict user access to a single image in an RBD pool? I know that I can use namespaces, so users can only see images with a given namespaces. However, these users will still be able to create new RBD images. Is it possible to somehow block users from creating RBD images and only work with the already existing ones?

2 1

Cephadm only scheduling, not orchestrating daemons
by Thomas Widhalm 14 Apr '23

14 Apr '23

Hi, As you might know, I have a problem with MDS not starting. During the investigation with your help I found another issue that might be related. I can plan to restart, redeploy, reconfigure services via cephadm or dashboard just as I want but services won't react. I only see the action to be scheduled but that's all. 2023-04-13T17:11:15.690698+0000 mgr.ceph04.qaexpv (mgr.74906907) 37184 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph05.pqxmvt 2023-04-13T17:11:20.746743+0000 mgr.ceph04.qaexpv (mgr.74906907) 37190 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph06.rrxmks 2023-04-13T17:11:24.971226+0000 mgr.ceph04.qaexpv (mgr.74906907) 37195 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph07.omdisd It's the same for other daemons/services. I changed placement rules, scheduled changes, failed mgr, even rebooted hosts. I even was desperate enough to delete files for services from hosts before rebooting hoping I could trigger a manual redeploy. All I see are the same MDS stuck in "error" state. I removed them via "ceph orch rm" but they are still there. When I reissue the command it fails saying that the service isn't there. "ceph orch ps" still lists them. mds.mds01.ceph03.xqwdjy ceph03 error 2d ago 2M - - <unknown> <unknown> <unknown> mds.mds01.ceph04.hcmvae ceph04 error 2d ago 2d - - <unknown> <unknown> <unknown> mds.mds01.ceph05.pqxmvt ceph05 error 2d ago 10M - - <unknown> <unknown> <unknown> mds.mds01.ceph06.rrxmks ceph06 error 2d ago 10w - - <unknown> <unknown> <unknown> mds.mds01.ceph07.omdisd ceph07 error 2d ago 3M - - <unknown> <unknown> <unknown> Any idea how I can get rid of them? Or redeploy them? Additionally I'm just in the middle of an upgrade. { "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Upgrade paused", "is_paused": true } I paused it on purpose to allow for manipulation of daemons. Cheers, Thomas

2 1

ceph 17.2.6 and iam roles (pr#48030)
by Christopher Durham 13 Apr '23

13 Apr '23

Hi, I see that this PR: https://github.com/ceph/ceph/pull/48030 made it into ceph 17.2.6, as per the change log at: https://docs.ceph.com/en/latest/releases/quincy/ That's great. But my scenario is as follows: I have two clusters set up as multisite. Because of the lack of replication for IAM roles, we have set things up so that roles on the primary 'manually' get replicated to the secondary site via a python script. Thus, if I create a role on the primary, add/delete users or buckets from said role, the role, including the AssumeRolePolicyDocument and policies, gets pushed to the replicated site. This has served us well for three years. With the advent of this fix, what should I do before I upgrade to 17.2.6 (currently on 17.2.5, rocky 8) I know that in my situation, roles of the same name have different RoleIDs on the two sites. What should I do before I upgrade? Possibilities that *could* happen if i dont rectify things as we upgrade: 1. The different RoleIDs lead to two roles of the same name on the replicated site, perhaps with the system unable to address/look at/modify either 2. Roles just don't get repiicated to the second site or other similar situations, all of which I want to avoid. Perhaps the safest thing to do is to remove all roles on the secondary site, upgrade, and then force a replication of roles (How would I *force* that for iAM roles if it is the correct answer?) Here is the original bug report: https://tracker.ceph.com/issues/57364 Thanks! -Chris

2 3

Live migrate RBD image with a client using it
by Work Ceph 13 Apr '23

13 Apr '23

Hello guys, We have been reading the docs, and trying to reproduce that process in our Ceph cluster. However, we always receive the following message: ``` librbd::Migration: prepare: image has watchers - not migrating rbd: preparing migration failed: (16) Device or resource busy ``` We tested with both RDB blocks mounted with kRBD, and with librbd via KVM/qEMU system. Both cases result in the same result. For KRBD, we understood that it is not supported right now, but for librbd it seems that it should be supported somehow. How do you guys handle those situations? We have the following use cases that might need an image to migrate between pools while the client is still consuming it. - RBD images that are consumed via the iSCSI gateways - RBD images mounted (rbd map) in hosts - RBD images used by KVM/Libvirt Does Ceph support a live migration of images between pools while the clients/consumers are still using those volumes? [1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/

5 5

RGW is slowly after the ops increase
by Louis Koo 13 Apr '23

13 Apr '23

check the ops for rgw: [root@node06 ceph]# ceph daemon /var/run/ceph/ceph-client.rgw.os.dsglczutvqsgowpz.a.13.93908447458760.asok objecter_requests| jq ".ops" | jq 'length' 8 list subdir with s5cmd: [root@node01 deeproute]# time ./s5cmd --endpoint-url=http://10.x.x.x:80 ls s3://mlp-data-warehouse/ads_prediction/ DIR prediction_scenes/ DIR test_pai/ real 0m1.125s user 0m0.007s sys 0m0.016s after the ops increase: [root@node06 ceph]# ceph daemon /var/run/ceph/ceph-client.rgw.os.dsglczutvqsgowpz.a.13.93908447458760.asok objecter_requests| jq ".ops" | jq 'length' 264 list subdir with s5cmd: [root@node01 deeproute]# time ./s5cmd --endpoint-url=http://10.x.x.x:80 ls s3://mlp-data-warehouse/ads_prediction/ DIR prediction_scenes/ DIR test_pai/ real 0m8.822s user 0m0.004s sys 0m0.019s and if the ops increase to more 2000, it needs more than 100s to list the subdir, why?

1 0

Osd crash, looks like something related to PG recovery.
by Louis Koo 12 Apr '23

12 Apr '23

{ "archived": "2023-04-13 02:23:50.948191", "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7f2ee8198ce0]", "pthread_kill()", "(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x48c) [0x563506e9934c]", "(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >)+0x23e) [0x563506e9973e]", "(PrimaryLogPG::scan_range(int, int, BackfillInterval*, ThreadPool::TPHandle&)+0x15a) [0x56350699a8da]", "(PrimaryLogPG::do_scan(boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x914) [0x56350699bd34]", "(PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x776) [0x56350699c826]", "(OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x563506823fc9]", "(ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x563506a82e78]", "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x5635068414c8]", "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x563506ebe2a4]", "(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563506ec1184]", "/lib64/libpthread.so.0(+0x81ca) [0x7f2ee818e1ca]", "clone()" ], "ceph_version": "16.2.10", "crash_id": "2023-04-12T16:53:45.988696Z_7e73aedd-3518-41f4-ae48-e5dbfe5750ec", "entity_name": "osd.74", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "1a2700ce6c68288739eb14ca1b2b5f49449c59a5baafbd1e71df3a4316e3bffe", "timestamp": "2023-04-12T16:53:45.988696Z", "utsname_hostname": "node03", "utsname_machine": "x86_64", "utsname_release": "3.10.0-1160.45.1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021" } releated issuse under rook: https://github.com/rook/rook/issues/11565

1 0

radosgw crash
by Louis Koo 12 Apr '23

12 Apr '23

{ "archived": "2023-04-09 01:22:40.755345", "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7f06dc1edce0]", "(boost::asio::detail::reactive_socket_service_base::start_op(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, int, boost::asio::detail::reactor_op*, bool, bool, bool)+0x126) [0x7f06e6cf84d6]", "(void boost::asio::detail::reactive_socket_service_base::async_receive<boost::asio::mutable_buffer, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> >(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, boost::asio::mutable_buffer const&, int, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >&, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> const&)+0x1fa) [0x7f06e6d0bd1a]", "(boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >::operator()(boost::system::error_code, unsigned long, bool)+0x17f) [0x7f06e6d39f8f]", "(boost::asio::detail::executor_op<boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x189) [0x7f06e6d3a7d9]", "(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x8d) [0x7f06e6d0793d]", "(void boost::asio::io_context::executor_type::dispatch<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, std::allocator<void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>&&, std::allocator<void> const&) const+0x9c) [0x7f06e6d07b3c]", "(void boost::asio::detail::strand_executor_service::dispatch<boost::asio::io_context::executor_type const, boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>, std::allocator<void> >(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl> const&, boost::asio::io_context::executor_type const&, boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>&&, std::allocator<void> const&)+0x2b6) [0x7f06e6d3a306]", "(boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffer, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x1a1) [0x7f06e6d3a571]", "(boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2) [0x7f06e6cfbad2]", "/lib64/libradosgw.so.2(+0x430376) [0x7f06e6cde376]", "/lib64/libstdc++.so.6(+0xc2ba3) [0x7f06db212ba3]", "/lib64/libpthread.so.0(+0x81ca) [0x7f06dc1e31ca]", "clone()" ], "ceph_version": "16.2.10", "crash_id": "2023-04-08T22:37:20.389262Z_88939939-522f-4b2c-a5fb-d4e49e9922a7", "entity_name": "client.rgw.os.dsglczutvqsgowpz.a", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "radosgw", "stack_sig": "2535cd0a26a2ffcc7ca223d416ebf3d4ea172eeec60026bb8b36b2c97ea787da", "timestamp": "2023-04-08T22:37:20.389262Z", "utsname_hostname": "node01", "utsname_machine": "x86_64", "utsname_release": "3.10.0-1160.45.1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021" }

1 0

rookcmd: failed to configure devices: failed to generate osd keyring: failed to get or create auth key for client.bootstrap-osd:
by knawaz＠juniper.net 12 Apr '23

12 Apr '23

I am deploying Ceph via ROOK on K8s Cluster with following version metric ceph-version=17.2.5-0 Ubuntu 20.04 Kernel 5.4.0-135-generic But getting following error, does ceph-version=17.2.5-0 tested with Ubuntu 20.04 running Kernel 5.4.0-135-generic. Where I can find compatibility metric? 2023-04-12 21:34:41.958118 I | cephclient: getting or creating ceph auth key "client.bootstrap-osd" 2023-04-12 21:34:41.958184 D | exec: Running command: ceph auth get-or-create-key client.bootstrap-osd mon allow profile bootstrap-osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json 2023-04-12 21:34:57.291895 C | rookcmd: failed to configure devices: failed to generate osd keyring: failed to get or create auth key for client.bootstrap-osd: failed get-or-create-key client.bootstrap-osd: exit status 1'

1 0

Upgrade from 17.2.5 to 17.2.6 stuck at MDS
by Thomas Widhalm 12 Apr '23

12 Apr '23

Hi, If you remember, I hit bug https://tracker.ceph.com/issues/58489 so I was very relieved when 17.2.6 was released and started to update immediately. But now I'm stuck again with my broken MDS. MDS won't get into up:active without the update but the update waits for them to get into up:active state. Seems like a deadlock / chicken-egg problem to me. Since I'm still relatively new to Ceph, could you help me? What I see when watching the update status: { "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Error: UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host ceph01 at addr (192.168.23.61)", "is_paused": false } (The offline host was one host that broke during the upgrade. I fixed that in the meantime and the update went on.) And in the log: 2023-04-10T19:23:48.750129+0000 mgr.ceph04.qaexpv [INF] Upgrade: Waiting for mds.mds01.ceph04.hcmvae to be up:active (currently up:replay) 2023-04-10T19:23:58.758141+0000 mgr.ceph04.qaexpv [WRN] Upgrade: No mds is up; continuing upgrade procedure to poke things in the right direction Please give me a hint what I can do. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

3 9

Pacific - not able to add more mons while setting up new cluster
by Boris Behrens 12 Apr '23

12 Apr '23

Hi, I am currently facing a rather strange issue and I have no idea where to look next. ceph.conf [global] fsid = ee52ffdf-5f03-4ff1-9c8d-1761a52d5c6a ms_bind_ipv6 = true ms_bind_ipv4 = false #mon_initial_members = 08f1ea96ef02, 08f1ea98b7c2, 08f1ea98ab3e mon_host = [fd00:2380:be45:43::10],[fd00:2380:be45:43::11],[fd00:2380:be45:43::12] auth_cluster_required = none auth_service_required = none auth_client_required = none public_network = fd00:2380:be45:43::/64 I can create the first mon with: monmaptool --create --add `hostname -s` fd00:2380:be45:43::10 --fsid ee52ffdf-5f03-4ff1-9c8d-1761a52d5c6a /tmp/monmap sudo -u ceph ceph-mon --cluster ceph --mkfs -i `hostname -s` --monmap /tmp/monmap systemctl enable --now ceph-mon@`hostname -s`.service systemctl enable --now ceph-mgr@`hostname -s`.service On the first host I can use the ceph command line as I would expect. Normally I then go to the next host and add it to the mons with ceph mon getmap -o /tmp/monmap sudo -u ceph ceph-mon --cluster ceph --mkfs -i `hostname -s` --monmap /tmp/monmap systemctl enable --now ceph-mon@`hostname -s`.service systemctl enable --now ceph-mgr@`hostname -s`.service But the ceph command hangs forever. Ping between the hosts work and when I telnet the remote mon I also get an answer. We do not have any authentication in place, and normally I would expect that it works this way. Any ideas what I can try? I had it working yesterday, but the restarted all nodes at the same time and then it just stopped and I never got it working again. Can it be switch config? I've added the debug output but it looks for as both systems communicate with each other but somehow it does not work: root@08f1ea98b7c2:~# ceph status --debug_ms=20 2023-04-12T15:55:29.253+0000 7f38bed9d700 2 Event(0x7f38c0069620 nevent=5000 time_id=1).set_owner center_id=0 owner=139881696843520 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).create_file_event create event started fd=4 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.add_event add event fd=4 cur_mask=0 add_mask=1 to 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).create_file_event create event end fd=4 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 stack operator() starting 2023-04-12T15:55:29.253+0000 7f38be59c700 2 Event(0x7f38c0060960 nevent=5000 time_id=1).set_owner center_id=1 owner=139881688450816 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=1).create_file_event create event started fd=7 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.add_event add event fd=7 cur_mask=0 add_mask=1 to 6 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=1).create_file_event create event end fd=7 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38be59c700 10 stack operator() starting 2023-04-12T15:55:29.253+0000 7f38bdd9b700 2 Event(0x7f38c0061950 nevent=5000 time_id=1).set_owner center_id=2 owner=139881680058112 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=1).create_file_event create event started fd=10 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.add_event add event fd=10 cur_mask=0 add_mask=1 to 9 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=1).create_file_event create event end fd=10 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 stack operator() starting 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- ready 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 Processor -- start 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- start start 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect v2:[fd00:2380:be45:43::12]:3300/0, creating connection and registering 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_NONE l=0)._connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect 0x7f38c0102340 v2:[fd00:2380:be45:43::12]:3300/0 v2:[fd00:2380:be45:43::12]:3300/0 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- connect_to v2:[fd00:2380:be45:43::12]:3300/0 new 0x7f38c0102340 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect v1:[fd00:2380:be45:43::10]:6789/0, creating connection and registering 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_NONE l=0)._connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect 0x7f38c0104dd0 v1:[fd00:2380:be45:43::10]:6789/0 v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- connect_to v1:[fd00:2380:be45:43::10]:6789/0 new 0x7f38c0104dd0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING l=0).process 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect v1:[fd00:2380:be45:43::12]:6789/0, creating connection and registering 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_NONE l=0)._connect 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=1).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- create_connect 0x7f38c0108c30 v1:[fd00:2380:be45:43::12]:6789/0 v1:[fd00:2380:be45:43::12]:6789/0 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 -- connect_to v1:[fd00:2380:be45:43::12]:6789/0 new 0x7f38c0108c30 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING l=0).process 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_keepalive 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).wakeup 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING l=0).process 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- --> v1:[fd00:2380:be45:43::10]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c010ca30 con 0x7f38c0104dd0 2023-04-12T15:55:29.253+0000 7f38bf59e700 15 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_message inline write is denied, reschedule m=0x7f38c010ca30 2023-04-12T15:55:29.253+0000 7f38bf59e700 10 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_keepalive 2023-04-12T15:55:29.253+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).wakeup 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- --> v1:[fd00:2380:be45:43::12]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c010dca0 con 0x7f38c0108c30 2023-04-12T15:55:29.253+0000 7f38bf59e700 15 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_message inline write is denied, reschedule m=0x7f38c010dca0 2023-04-12T15:55:29.253+0000 7f38bf59e700 1 -- --> v2:[fd00:2380:be45:43::12]:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c010eef0 con 0x7f38c0102340 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event started fd=12 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bf59e700 5 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).send_message enqueueing message m=0x7f38c010eef0 type=5 mon_getmap magic: 0 v1 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=0 add_mask=1 to 6 2023-04-12T15:55:29.253+0000 7f38bf59e700 15 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).send_message inline write is denied, reschedule m=0x7f38c010eef0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event end fd=12 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38be59c700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event started fd=13 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38be59c700 10 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process nonblock connect inprogress 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event started fd=12 mask=2 original mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.add_event add event fd=13 cur_mask=0 add_mask=1 to 3 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=1 add_mask=2 to 6 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event end fd=13 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.253+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=2).create_file_event create event end fd=12 mask=2 current mask is 3 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event started fd=14 mask=1 original mask is 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.add_event add event fd=14 cur_mask=0 add_mask=1 to 9 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process nonblock connect inprogress 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event end fd=14 mask=1 current mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event started fd=13 mask=2 original mask is 1 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.add_event add event fd=13 cur_mask=1 add_mask=2 to 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 Event(0x7f38c0069620 nevent=5000 time_id=2).create_file_event create event end fd=13 mask=2 current mask is 3 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).handle_write 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).write_event 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).process nonblock connect inprogress 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event started fd=14 mask=2 original mask is 1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.add_event add event fd=14 cur_mask=1 add_mask=2 to 9 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 NetHandler reconnect reconnect: 111 Connection refused 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 Event(0x7f38c0061950 nevent=5000 time_id=2).create_file_event create event end fd=14 mask=2 current mask is 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 1 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:[fd00:2380:be45:43::12]:6789/0 2023-04-12T15:55:29.253+0000 7f38be59c700 20 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process 2023-04-12T15:55:29.253+0000 7f38bed9d700 2 -- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process connection refused! 2023-04-12T15:55:29.253+0000 7f38be59c700 10 NetHandler reconnect reconnect: 111 Connection refused 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).handle_write 2023-04-12T15:55:29.253+0000 7f38be59c700 1 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:[fd00:2380:be45:43::12]:3300/0 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).fault 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).write_event 2023-04-12T15:55:29.253+0000 7f38be59c700 2 -- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING_RE l=0).process connection refused! 2023-04-12T15:55:29.253+0000 7f38bed9d700 20 EpollDriver.del_event del event fd=13 cur_mask=3 delmask=3 to 3 2023-04-12T15:55:29.253+0000 7f38bed9d700 5 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state 2023-04-12T15:55:29.253+0000 7f38bed9d700 5 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security 2023-04-12T15:55:29.253+0000 7f38bed9d700 10 --1- >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).fault waiting 0.200000 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 EpollDriver.del_event del event fd=14 cur_mask=3 delmask=2 to 9 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTING_RE l=0).process connect successfully, ready to send banner 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).read_event 2023-04-12T15:55:29.253+0000 7f38be59c700 10 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=START_CONNECT pgs=0 cs=0 l=0).send_client_banner 2023-04-12T15:55:29.253+0000 7f38be59c700 20 EpollDriver.del_event del event fd=12 cur_mask=3 delmask=3 to 6 2023-04-12T15:55:29.253+0000 7f38be59c700 5 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_recv_state 2023-04-12T15:55:29.253+0000 7f38be59c700 5 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_security 2023-04-12T15:55:29.253+0000 7f38be59c700 1 --2- >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 9 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING pgs=0 cs=0 l=0).handle_client_banner_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING pgs=0 cs=0 l=0).handle_client_banner_write connect write banner done: v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).wait_server_banner 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=281 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=281 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify connect read peer addr v1:[fd00:2380:be45:43::10]:6789/0 on socket 14 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify connect peer addr for me is v1:[fd00:2380:be45:43::11]:44160/0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 1 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:[fd00:2380:be45:43::10]:6789/0 says I am v1:[fd00:2380:be45:43::11]:44160/0 (socket says [fd00:2380:be45:43::11]:44160) 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 learned_addr had no addrs 2023-04-12T15:55:29.253+0000 7f38bdd9b700 1 -- [fd00:2380:be45:43::11]:0/2007947341 learned_addr learned my addr [fd00:2380:be45:43::11]:0/2007947341 (peer_addr_for_me v1:[fd00:2380:be45:43::11]:0/0) 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 136 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_my_addr_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_my_addr_write connect sent my addr v1:[fd00:2380:be45:43::11]:0/2007947341 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).send_connect_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).send_connect_message connect sending gseq=1 cseq=0 proto=15 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 33 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_message_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_message_write connect wrote (self +) cseq, waiting for reply 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).wait_connect_reply 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=26 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read continue len=26 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 connect got reply tag 13 connect_seq 1 global_seq 52 proto 15 flags 1 features 4540138314316775423 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 got CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).wait_ack_seq 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read start len=8 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_ack_seq r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_ack_seq got newly_acked_seq 0 vs out_seq 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).discard_requeued_up_to 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send sent bytes 8 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_in_seq_write r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_in_seq_write send in_seq done 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).client_ready 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=52 cs=1 l=1).client_ready connect success 1, lossy = 1, features 4540138314316775423 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=52 cs=1 l=1).client_ready no authorizer, clearing session_security 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_event 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message m auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c010ca30 auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending message type=17 src client.? front=60 data=0 off 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 1 0x7f38c010ca30 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 144 remaining bytes 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 0x7f38c010ca30 done. 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 15 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_keepalive2_ack r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_keepalive2_ack got KEEPALIVE_ACK 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=18 src mon.0 front=24 data=0 off 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=24 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 24 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 24 + 0 + 0 byte message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.253+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 1 0x7f38c010ca30 auth_reply(proto 1 0 (0) Success) v1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38c010ca30 prio 196 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.253+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 1 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (unknown 1154771820 0 0) 0x7f38c010ca30 con 0x7f38c0104dd0 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.253+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 msgr2=0x7f38c0104790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.257+0000 7f38bd59a700 5 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).discard_out_queue discard mon_getmap magic: 0 v1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.257+0000 7f38be59c700 5 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_recv_state (warped) reseting crypto handlers 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 legacy=0x7f38c010b0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-04-12T15:55:29.257+0000 7f38be59c700 5 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).reset_security 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 2 0x7f38b4014b00 mon_map magic: 0 v1 2023-04-12T15:55:29.257+0000 7f38bd59a700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38b4014b00 prio 196 2023-04-12T15:55:29.257+0000 7f38bd59a700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).discard_out_queue discard 0x7f38c010dca0 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c010f080 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bd59a700 15 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c010f080 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 24 to dispatch throttler 239/104857600 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38c010ca30 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_event 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 2 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b4014b00 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message m mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c010f080 mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending message type=15 src client.? front=58 data=0 off 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 2 0x7f38c010f080 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 133 remaining bytes 0 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38b4014b00 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).write_message sending 0x7f38c010f080 done. 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=62 src mon.0 front=4 data=0 off 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=4 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 4 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 4 + 0 + 0 byte message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 3 0x7f38b4014df0 config(0 keys) v1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38b4014df0 prio 196 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 3 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b4014df0 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 4 to dispatch throttler 4/104857600 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38b4014df0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_MESSAGE pgs=52 cs=1 l=1).throttle_message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_BYTES pgs=52 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=52 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_front 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_MESSAGE_FRONT pgs=52 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.257+0000 7f38bdd9b700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=READ_FOOTER_AND_DISPATCH pgs=52 cs=1 l=1). rx mon.0 seq 4 0x7f38b40151a0 mon_map magic: 0 v1 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 queue 0x7f38b40151a0 prio 196 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).wait_message 2023-04-12T15:55:29.257+0000 7f38bd59a700 1 -- [fd00:2380:be45:43::11]:0/2007947341 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 4 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b40151a0 con 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.257+0000 7f38bd59a700 10 -- [fd00:2380:be45:43::11]:0/2007947341 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.257+0000 7f38bd59a700 20 -- [fd00:2380:be45:43::11]:0/2007947341 done calling dispatch on 0x7f38b40151a0 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 legacy=0x7f38c0107290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=OPENED pgs=52 cs=1 l=1).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CLOSED pgs=52 cs=1 l=1).reset_recv_state (warped) reseting security handlers 2023-04-12T15:55:29.257+0000 7f38bdd9b700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CLOSED pgs=52 cs=1 l=1).reset_security 2023-04-12T15:55:29.257+0000 7f38bdd9b700 20 EpollDriver.del_event del event fd=14 cur_mask=1 delmask=3 to 9 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown [fd00:2380:be45:43::11]:0/2007947341 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 Processor -- stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections mark down v1:[fd00:2380:be45:43::12]:6789/0 0x7f38c0108c30 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::12]:6789/0 conn(0x7f38c0108c30 0x7f38c010b0f0 :-1 s=CLOSED pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections mark down v1:[fd00:2380:be45:43::10]:6789/0 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0104dd0 0x7f38c0107290 :-1 s=CLOSED pgs=52 cs=1 l=1).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections mark down v2:[fd00:2380:be45:43::12]:3300/0 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 --2- [fd00:2380:be45:43::11]:0/2007947341 >> v2:[fd00:2380:be45:43::12]:3300/0 conn(0x7f38c0102340 0x7f38c0104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0104dd0 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0108c30 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 msgr2=0x7f38c00632e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 2 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).stop 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).reset_recv_state 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=NONE pgs=0 cs=0 l=0).discard_out_queue started 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers 2023-04-12T15:55:29.257+0000 7f38bed9d700 5 --1- [fd00:2380:be45:43::11]:0/2007947341 >> [fd00:2380:be45:43::11]:0/2007947341 conn(0x7f38c0062e70 0x7f38c00632e0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 wait: waiting for dispatch queue 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 wait: dispatch queue is stopped 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections 2023-04-12T15:55:29.257+0000 7f38bf59e700 5 -- [fd00:2380:be45:43::11]:0/2007947341 shutdown_connections delete 0x7f38c0062e70 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0061950 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- [fd00:2380:be45:43::11]:0/2007947341 wait: done. 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/2007947341 wait complete. 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- ready 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 Processor -- start 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0069620 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- start start 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- create_connect v1:[fd00:2380:be45:43::10]:6789/0, creating connection and registering 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_NONE l=1)._connect 2023-04-12T15:55:29.257+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=3).wakeup 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- create_connect 0x7f38c0102340 v1:[fd00:2380:be45:43::10]:6789/0 v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 -- connect_to v1:[fd00:2380:be45:43::10]:6789/0 new 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).send_keepalive 2023-04-12T15:55:29.257+0000 7f38bf59e700 1 -- --> v1:[fd00:2380:be45:43::10]:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c0067e30 con 0x7f38c0102340 2023-04-12T15:55:29.257+0000 7f38bf59e700 15 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).send_message inline write is denied, reschedule m=0x7f38c0067e30 2023-04-12T15:55:29.257+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING l=1).process 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event started fd=12 mask=1 original mask is 0 2023-04-12T15:55:29.257+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=0 add_mask=1 to 6 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event end fd=12 mask=1 current mask is 1 2023-04-12T15:55:29.257+0000 7f38be59c700 10 NetHandler reconnect reconnect: 114 Operation already in progress 2023-04-12T15:55:29.257+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).process nonblock connect inprogress 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event started fd=12 mask=2 original mask is 1 2023-04-12T15:55:29.257+0000 7f38be59c700 20 EpollDriver.add_event add event fd=12 cur_mask=1 add_mask=2 to 6 2023-04-12T15:55:29.257+0000 7f38be59c700 20 Event(0x7f38c0060960 nevent=5000 time_id=4).create_file_event create event end fd=12 mask=2 current mask is 3 2023-04-12T15:55:29.257+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).handle_write 2023-04-12T15:55:29.257+0000 7f38be59c700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).write_event 2023-04-12T15:55:29.257+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).process 2023-04-12T15:55:29.257+0000 7f38be59c700 20 EpollDriver.del_event del event fd=12 cur_mask=3 delmask=2 to 6 2023-04-12T15:55:29.257+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTING_RE l=1).process connect successfully, ready to send banner 2023-04-12T15:55:29.257+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).read_event 2023-04-12T15:55:29.257+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=START_CONNECT pgs=0 cs=0 l=1).send_client_banner 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 9 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING pgs=0 cs=0 l=1).handle_client_banner_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING pgs=0 cs=0 l=1).handle_client_banner_write connect write banner done: v1:[fd00:2380:be45:43::10]:6789/0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).wait_server_banner 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=281 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=281 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify connect read peer addr v1:[fd00:2380:be45:43::10]:6789/0 on socket 12 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify connect peer addr for me is v1:[fd00:2380:be45:43::11]:44170/0 2023-04-12T15:55:29.261+0000 7f38be59c700 1 --1- >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:[fd00:2380:be45:43::10]:6789/0 says I am v1:[fd00:2380:be45:43::11]:44170/0 (socket says [fd00:2380:be45:43::11]:44170) 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 learned_addr had no addrs 2023-04-12T15:55:29.261+0000 7f38be59c700 1 -- [fd00:2380:be45:43::11]:0/4275451039 learned_addr learned my addr [fd00:2380:be45:43::11]:0/4275451039 (peer_addr_for_me v1:[fd00:2380:be45:43::11]:0/0) 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 136 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_my_addr_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_my_addr_write connect sent my addr v1:[fd00:2380:be45:43::11]:0/4275451039 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).send_connect_message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).send_connect_message connect sending gseq=1 cseq=0 proto=15 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 33 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_message_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_message_write connect wrote (self +) cseq, waiting for reply 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).wait_connect_reply 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=26 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=26 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 connect got reply tag 13 connect_seq 1 global_seq 53 proto 15 flags 1 features 4540138314316775423 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_2 got CEPH_MSGR_TAG_SEQ, reading acked_seq and writing in_seq 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).wait_ack_seq 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_ack_seq r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 2 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_ack_seq got newly_acked_seq 0 vs out_seq 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).discard_requeued_up_to 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 8 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_in_seq_write r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_in_seq_write send in_seq done 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).client_ready 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=53 cs=1 l=1).client_ready connect success 1, lossy = 1, features 4540138314316775423 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=53 cs=1 l=1).client_ready no authorizer, clearing session_security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c0067e30 auth(proto 0 30 bytes epoch 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=17 src client.? front=60 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 1 0x7f38c0067e30 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 144 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c0067e30 done. 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 15 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=8 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_keepalive2_ack r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_keepalive2_ack got KEEPALIVE_ACK 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=18 src mon.0 front=24 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=24 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 24 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 24 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 1 0x7f38c0067e30 auth_reply(proto 1 0 (0) Success) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38c0067e30 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 1 ==== auth_reply(proto 1 0 (0) Success) v1 ==== 24+0+0 (unknown 2435360628 0 0) 0x7f38c0067e30 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0068070 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38aeffd700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c0068070 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 24 to dispatch throttler 239/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38c0067e30 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 2 0x7f38b0025be0 mon_map magic: 0 v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38b0025be0 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 2 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b0025be0 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c0068070 mon_subscribe({config=0+,monmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38b0025be0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=15 src client.? front=58 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 2 0x7f38c0068070 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 133 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c0068070 done. 2023-04-12T15:55:29.261+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c0068350 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38bf59e700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c0068350 2023-04-12T15:55:29.261+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_subscribe({mgrmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c0068350 mon_subscribe({mgrmap=0+}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=15 src client.4108 front=39 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 3 0x7f38c0068350 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 114 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c006b410 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c0068350 done. 2023-04-12T15:55:29.261+0000 7f38bf59e700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f38c006b410 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_subscribe({osdmap=0}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f38c006b410 mon_subscribe({osdmap=0}) v3 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=15 src client.4108 front=39 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 4 0x7f38c006b410 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 114 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f38c006b410 done. 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).process 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read continue len=1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=62 src mon.0 front=4 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=4 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 4 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 4 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 3 0x7f38b0025f40 config(0 keys) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38b0025f40 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 3 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38b0025f40 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message process tag 7 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message begin MSG 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 4 to dispatch throttler 4/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38b0025f40 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=53 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got MSG header 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message_header got envelope type=4 src mon.0 front=215 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_MESSAGE pgs=53 cs=1 l=1).throttle_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_BYTES pgs=53 cs=1 l=1).throttle_bytes 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=THROTTLE_DISPATCH_QUEUE pgs=53 cs=1 l=1).throttle_dispatch_queue 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_front 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).handle_message_front got front 215 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_middle 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data_prepare 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_data msg_left=0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_MESSAGE_FRONT pgs=53 cs=1 l=1).read_message_footer 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=21 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer r=0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer aborted = 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer got 215 + 0 + 0 byte message 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1).handle_message_footer no session security set 2023-04-12T15:55:29.261+0000 7f38be59c700 5 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=READ_FOOTER_AND_DISPATCH pgs=53 cs=1 l=1). rx mon.0 seq 4 0x7f38b0026270 mon_map magic: 0 v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 queue 0x7f38b0026270 prio 196 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).wait_message 2023-04-12T15:55:29.261+0000 7f38be59c700 20 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read start len=1 2023-04-12T15:55:29.261+0000 7f38aeffd700 1 -- [fd00:2380:be45:43::11]:0/4275451039 <== mon.0 v1:[fd00:2380:be45:43::10]:6789/0 4 ==== mon_map magic: 0 v1 ==== 215+0+0 (unknown 253804068 0 0) 0x7f38b0026270 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38aeffd700 10 -- [fd00:2380:be45:43::11]:0/4275451039 dispatch_throttle_release 215 to dispatch throttler 215/104857600 2023-04-12T15:55:29.261+0000 7f38aeffd700 20 -- [fd00:2380:be45:43::11]:0/4275451039 done calling dispatch on 0x7f38b0026270 2023-04-12T15:55:29.261+0000 7f38bf59e700 1 -- [fd00:2380:be45:43::11]:0/4275451039 --> v1:[fd00:2380:be45:43::10]:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c004f50 con 0x7f38c0102340 2023-04-12T15:55:29.261+0000 7f38bf59e700 15 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_message inline write is denied, reschedule m=0x7f389c004f50 2023-04-12T15:55:29.261+0000 7f38bf59e700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message m mon_command({"prefix": "get_command_descriptions"} v 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).prepare_send_message encoding features 4540138314316775423 0x7f389c004f50 mon_command({"prefix": "get_command_descriptions"} v 0) v1 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message no session security 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending message type=50 src client.4108 front=80 data=0 off 0 2023-04-12T15:55:29.261+0000 7f38be59c700 20 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 5 0x7f389c004f50 2023-04-12T15:55:29.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 155 remaining bytes 0 2023-04-12T15:55:29.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_message sending 0x7f389c004f50 done. 2023-04-12T15:55:32.261+0000 7f38adffb700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_keepalive 2023-04-12T15:55:32.261+0000 7f38adffb700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:32.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:32.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:32.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:32.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 9 remaining bytes 0 2023-04-12T15:55:42.261+0000 7f38adffb700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).send_keepalive 2023-04-12T15:55:42.261+0000 7f38adffb700 20 Event(0x7f38c0060960 nevent=5000 time_id=5).wakeup 2023-04-12T15:55:42.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).handle_write 2023-04-12T15:55:42.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).write_event 2023-04-12T15:55:42.261+0000 7f38be59c700 10 --1- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 0x7f38c0067590 :-1 s=OPENED pgs=53 cs=1 l=1).append_keepalive_or_ack 2023-04-12T15:55:42.261+0000 7f38be59c700 10 -- [fd00:2380:be45:43::11]:0/4275451039 >> v1:[fd00:2380:be45:43::10]:6789/0 conn(0x7f38c0102340 legacy=0x7f38c0067590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send sent bytes 9 remaining bytes 0 ^CInterrupted -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 1

pacific v16.2.1 (hot-fix) QE Validation status
by Yuri Weinstein 12 Apr '23

12 Apr '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59426#note-3 Release Notes - TBD Seeking approvals/reviews for: smoke - Josh approved? orch - Adam King approved? (there are infrastructure issues in the runs, but we want to release this ASAP) Thx YuriW

3 2

RBD snapshot mirror syncs all snapshots
by Andreas Teuchert 12 Apr '23

12 Apr '23

Hello, I setup two-way snapshot-based RBD mirroring between two Ceph clusters. After enabling mirroring for an image that already had regular snapshots independently from RBD mirror on the source cluster, the image and all snapshots were synced to the destination cluster. Is there a way to avoid having all snapshots being synced? We only need the latest version of the image on the destination cluster and the snapshots add around 200% disk space overhead on average. Best regards, Andreas

1 0

Ceph Leadership Team Meeting, 2023-04-12 Minutes
by Patrick Donnelly 12 Apr '23

12 Apr '23

Hi folks, Today we discussed: - Just short of 1 exabyte of Ceph storage reported to Telemetry. Telemetry's data is public and viewable at: https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?orgId=1 If your cluster is not reporting to Telemetry, please consider it! :) - A request from the Ceph Foundation Board to begin tracking component (e.g. CephFS) roadmaps in docs (or somewhere else appropriate). Concurrently, leads may also begin sending out status updates on a ~quarterly basis. To be discussed further. - Cephalocon schedule is available: https://ceph2023.sched.com/ - A regression was reported for the exporter in 17.2.6: https://github.com/ceph/ceph/pull/50718#issuecomment-1503376925 A follow-up hotfix/announcement is planned. - Next week's meeting is canceled due to Cephalocon/travel. Meeting minutes available here as always: https://pad.ceph.com/p/clt-weekly-minutes -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: avg apply latency went up after update from octopus to pacific
by Jan-Tristan Kruse 12 Apr '23

12 Apr '23

Hi, we solved our latency-problems in two clusters by redeploying all OSDs. Since then, we did not reencounter the problems described. Greetings, Jan

2 1

[RGW] Rebuilding a non master zone
by Gilles Mocellin 12 Apr '23

12 Apr '23

Hello cephers ! As I was asking in another thread ([RGW] Rebuilding a non master zone), I try to find the best way to rebuild a zone in a multisite config. The goal is to get rid of remaining Large OMAP objects. The simplest way, as I can rely only on the primary zone, is to : - remove the zone from the zonegroup - delete the pools - recreate them by restarting radosgw instances - add the zone in the zonegroup Is there a simpler way ? One question I already asked was : can I delete only index and data pools (ZONE.rgw.buckets.data and ZONE.rgw.buckets.index), the multisite config (zones and zonegroup) should be still there, and perhaps it will just resync (or I can force a sync init) ? Another thing, about ceph-ansible, which I still use. I configure my zones with the multisite config of ceph-ansible, but one thing bothers me : it defines the zone endpoints on each radosgw instances, even with the loadbalancer configured. So, do you think I'd better not configure multisite with ceph-ansible, and do it manually ? Is there a way that will make things easier to migrate to cephadm ?

1 0

Pacific dashboard: unable to get RGW information
by Michel Jouvin 12 Apr '23

12 Apr '23

Hi, Our cluster is running Pacific 16.2.10. We have a problem using the dashboard to display information about RGWs configured in the cluster. When clicking on "Object Gateway", we get an error 500. Looking in the mgr logs, I found that the problem is that the RGW is accessed by its IP address rather than its name. As the RGW has SSL enabled, the certificate cannot be matched against the IP address. I digged into the configuration but I was not able to identify where an IP address rather than a name was used (I checked in particular the zonegroup parameters and names are used to define endpoints). Did I make something wrong in the configuration or is it a know issue when using SSL-enabled RGW? Best regards, Michel

4 7

Module 'cephadm' has failed: invalid literal for int() with base 10:
by Duncan M Tooke 12 Apr '23

12 Apr '23

Hi, Our Ceph cluster is in an error state with the message: # ceph status cluster: id: 58140ed2-4ed4-11ed-b4db-5c6f69756a60 health: HEALTH_ERR Module 'cephadm' has failed: invalid literal for int() with base 10: '352.broken' This happened after trying to re-add an OSD which had failed. Adopting it back in to the Ceph failed because a directory was causing problems in /var/lib/ceph/{cephid}/osd.352. To re-add the OSD I renamed it to osd.352.broken (rather than delete it), re-ran the command and then everything worked perfectly. Then 5 minutes later the ceph orchestrator went into "HEALTH_ERR" I've removed that directory, but "cephadm" isn't cleaning up after itself. Does anyone know if there's a way I can clear the cache for this directory it's tried to inventory and failed? Thanks, Duncan -- Dr Duncan Tooke | Research Cluster Administrator Centre for Computational Biology, Weatherall Institute of Molecular Medicine, University of Oxford, OX3 9DS www.imm.ox.ac.uk<http://www.imm.ox.ac.uk>

2 2

radosgw-admin bucket stats doesn't show real num_objects and size
by viplanghe6＠gmail.com 11 Apr '23

11 Apr '23

The radosgw-admin bucket stats show there are 209266 objects in this bucket, but it included failed multiparts, so that make the size parameter is also wrong. When I use boto3 to count objects, the bucket only has 209049 objects. The only solution I can find is to use lifecycle to clean these failed multiparts, but in production, the client will decide to use lifecycle or not? So are there any way to exclude the failed multiparts in bucket statistic? Does Ceph allow to set auto clean failed multiparts globally? Thanks! "usage": { "rgw.main": { "size": 593286801276, "size_actual": 593716080640, "size_utilized": 593286801276, "size_kb": 579381642, "size_kb_actual": 579800860, "size_kb_utilized": 579381642, "num_objects": 209266 }

3 2

naming the S release
by Josh Durgin 11 Apr '23

11 Apr '23

With the Reef dev cycle closing, it's time to think about S and future releases. There are a bunch of options for S already, add a +1 or a new option to this etherpad, and we'll see what has the most votes next week: https://pad.ceph.com/p/s Josh

1 0

Ceph Object Gateway and lua scripts
by Thomas Bennett 11 Apr '23

11 Apr '23

Hi, We're currently testing out lua scripting in the Ceph Object Gateway (Radosgw). Ceph version: 17.2.5 We've tried a simple experiment with the simple lua script which is based on the documentation (see fixed width text below). However, the issue we're having is that we can't find the log messages anywhere. We've searched the entire jourrnalctl database as well as raised the debug level on the radosgw by setting debug_rgw to 20 on the running daemon. Any help welcome :) function print_object(msg, object) RGWDebugLog(" Title: " .. msg) RGWDebugLog(" Name: " .. object.Name) RGWDebugLog(" Instance: " .. object.Instance) RGWDebugLog(" Id: " .. object.Id) RGWDebugLog(" Size: " .. object.Size) RGWDebugLog(" MTime: " .. object.MTime) end RGWDebugLog("This is a log message!") Request.Log() if Request.CopyFrom then print_object("copy from", Request.CopyFrom.Object) if Request.CopyFrom.Object then print_object("copy from-object" ,Request.CopyFrom.Object) end end if Request.Object then print_object("Object" ,Request.Object) end Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.

2 3

RGW don't use .rgw.root multisite configuration
by Guillaume Morin 11 Apr '23

11 Apr '23

Hello, for my multisite configuration, i create and use an root pool other than .rgw.root to store realm and zone configuration with the follow option: rgw_realm_root_pool=myzone.rgw.root rgw_zonegroup_root_pool=myzone.rgw.root rgw_zone_root_pool=myzone.rgw.root But , i can see during the syncronize with my secondary zone (real only) , radosgw need to read and write period in .rgw.root. Is there an option or other to avoid read and write on the root pool .rgw.root ? Is it mandatory ? Pacific regards Guillaume

3 2

Announcing go-ceph v0.21.0
by John Mulligan 11 Apr '23

11 Apr '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.21.0 Changes include additions to the rbd, cephfs, and cephfs/admin packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Why is my cephfs almostfull?
by Jorge Garcia 11 Apr '23

11 Apr '23

We have a ceph cluster with a cephfs filesystem that we use mostly for backups. When I do a "ceph -s" or a "ceph df", it reports lots of space: data: pools: 3 pools, 4104 pgs objects: 1.09 G objects, 944 TiB usage: 1.5 PiB used, 1.0 PiB / 2.5 PiB avail GLOBAL: SIZE AVAIL RAW USED %RAW USED 2.5 PiB 1.0 PiB 1.5 PiB 59.76 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS cephfs_data 2 944 TiB 87.63 133 TiB 880988429 cephfs_metadata 3 128 MiB 0 62 TiB 206535313 .rgw.root 4 0 B 0 62 TiB 0 The whole thing consists of 2 pools: metadata (regular default replication) and data (erasure k:5 m:2). The global raw space reports 2.5 PiB total, with 1.0 PiB still available. But, when the ceph filesystem is mounted, it only reports 1.1 PB total, and the filesystem is almost full: Filesystem Size Used Avail Use% Mounted on x.x.x.x:yyyy:/ 1.1P 944T 134T 88% /backups So, where is the rest of my space? Or what am I missing? Thanks!

3 2

Disks are filling up even if there is not a single placement group on them
by Michal Strnad 11 Apr '23

11 Apr '23

Hi all, Our SSD disks are filling up, even if there is not a single placement group on them. How is that possible please? ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 1997 ssd 1.15269 1.00000 1.2 TiB 943 GiB 942 GiB 6 KiB 1.5 GiB 237 GiB 79.90 1.71 0 up Cluster uses Pacific 16.2.7. Thank you Michal

3 4

ceph.v17 multi-mds ephemeral directory pinning: cannot set or retrieve extended attribute
by Ulrich Pralle 10 Apr '23

10 Apr '23

Hi, we are using ceph version 17.2.5 on Ubuntu 22.04.1 LTS. We deployed multi-mds (max_mds=4, plus standby-replay mds). Currently we statically directory-pinned our user home directories (~50k). The cephfs' root directory is pinned to '-1', ./homes is pinned to "0". All user home directories below ./homes/ are pinned to -1, 1, 2, or 3 depending on a simple hash algorithm. Cephfs is provided to our users as samba/cifs (clustered samba,ctdb). We want to try ephemeral directory pinning. We can successfully set the extended attribute "ceph.dir.pin.distributed" with setfattr(1), but cannot retrieve its setting afterwards.: # setfattr -n ceph.dir.pin.distributed -v 1 ./units # getfattr -n ceph.dir.pin.distributed ./units ./units: ceph.dir.pin.distributed: No such attribute strace setfattr reports success on setxattr setxattr("./units", "ceph.dir.pin.distributed", "1", 1, 0) = 0 strace getfattr reports lstat("./units", {st_mode=S_IFDIR|0751, st_size=1, ...}) = 0 getxattr("./units", "ceph.dir.pin.distributed", NULL, 0) = -1 ENODATA (No data available) The file system is mounted rw,noatime,,name=<omitted>,mds_namespace=<omitted>.acl,recover_session=clean. The cephfs mds caps are "allow rwps". "./units" has a ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=fs_data_units" Ubuntu's setfattr is version 2.4.48. Defining other cephfs extend attributes (like ceph.dir.pin, ceph.quota.max_bytes, etc.) works as expected. What are we missing? Should we clear all static directory pinnings in advance? Are there any experience with ephemeral directory pinning? Or should one refrain from multi-mds at all? Kind regards Uli Pralle

2 1

v17.2.6 Quincy released
by Yuri Weinstein 10 Apr '23

10 Apr '23

We're happy to announce the 6th backport release in the Quincy series. https://ceph.io/en/news/blog/2023/v17-2-6-quincy-released/ Notable Changes --------------- * `ceph mgr dump` command now outputs `last_failure_osd_epoch` and `active_clients` fields at the top level. Previously, these fields were output under `always_on_modules` field. * telemetry: Added new metrics to the 'basic' channel to report per-pool bluestore compression metrics. See a sample report with `ceph telemetry preview`. Opt-in with `ceph telemetry on`. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.6.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: d7ff0d10654d2280e08f1ab989c7cdf3064446a5

1 0

quincy v17.2.6 QE Validation status
by Yuri Weinstein 07 Apr '23

07 Apr '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59070#note-1 Release Notes - TBD The reruns were in the queue for 4 days because of some slowness issues. The core team (Neha, Radek, Laura, and others) are trying to narrow down the root cause. Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to test and merge at least one PR https://github.com/ceph/ceph/pull/50575 for the core) rgw - Casey fs - Venky (the fs suite has an unusually high amount of failed jobs, any reason to suspect it in the observed slowness?) orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - Laura is looking into failures upgrade/pacific-x - Laura is looking into failures upgrade/quincy-p2p - Laura is looking into failures client-upgrade-octopus-quincy-quincy - missing packages, Adam Kraitman is looking into it powercycle - Brad ceph-volume - needs a rerun on merged https://github.com/ceph/ceph-ansible/pull/7409 Please reply to this email with approval and/or trackers of known issues/PRs to address them. Also, share any findings or hypnosis about the slowness in the execution of the suite. Josh, Neha - gibba and LRC upgrades pending major suites approvals. RC release - pending major suites approvals. Thx YuriW

14 27

Cephadm - Error ENOENT: Module not found
by elia.oggian＠cscs.ch 06 Apr '23

06 Apr '23

Hello, After a successful upgrade of a Ceph cluster from 16.2.7 to 16.2.11, I needed to downgrade it back to 16.2.7 as I found an issue with the new version. I expected that running the downgrade with:`ceph orch upgrade start --ceph-version 16.2.7` should have worked fine. However, it blocked right after the downgrade of the first MGR daemon. In fact, the downgraded daemon is not able to use the cephadm module anymore. Any `ceph orch` command fails with the following error: ``` $ ceph orch ps Error ENOENT: Module not found ``` And the downgrade process is therefore blocked. These are the logs of the MGR when issuing the command: ``` Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.557+0000 7f828fe8c700 0 log_channel(audit) log [DBG] : from='client.3136173 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.list_daemons(*(None, None), **{'daemon_id': None, 'host': None, 'refresh': False}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 0 [orchestrator DEBUG root] _oremote orchestrator -> cephadm.get_feature_set(*(), **{}) Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 no module 'cephadm' Mar 28 12:13:15 astano03 ceph-c57586c4-8e44-11eb-a116-248a07aa8d2e-mgr-astano03-qtzccn[2232770]: debug 2023-03-28T10:13:15.558+0000 7f829068d700 -1 mgr.server reply reply (2) No such file or directory Module not found ``` Other interesting MGR logs are: ``` 2023-03-28T11:05:59.519+0000 7fcd16314700 4 mgr get_store get_store key: mgr/cephadm/upgrade_state 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Failed to construct class in 'cephadm' 2023-03-28T11:05:59.519+0000 7fcd16314700 -1 mgr load Traceback (most recent call last): e "/usr/share/ceph/mgr/cephadm/module.py", line 450, in __init__ elf.upgrade = CephadmUpgrade(self) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 111, in __init__ elf.upgrade_state: Optional[UpgradeState] = UpgradeState.from_json(json.loads(t)) e "/usr/share/ceph/mgr/cephadm/upgrade.py", line 92, in from_json eturn cls(**c) rror: __init__() got an unexpected keyword argument 'daemon_types' 2023-03-28T11:05:59.521+0000 7fcd16314700 -1 mgr operator() Failed to run module in active mode ('cephadm') ``` Which seem to relate to the new feature of staggered upgrades. Please note that before, everything was working fine with version 16.2.7. I am currently stuck in this situation with only one MGR daemon on version 16.2.11 which is the only one still working fine: ``` [root@astano01 ~]# ceph orch ps | grep mgr mgr.astano02.mzmewn astano02 *:8443,9283 running (5d) 43s ago 2y 455M - 16.2.11 7a63bce27215 e2d7806acf16 mgr.astano03.qtzccn astano03 *:8443,9283 running (3m) 22s ago 95m 383M - 16.2.7 463ec4b1fdc0 cc0d88864fa1 ``` Does anyone already faced this issue or knows how can I make the 16.2.7 MGR load the cephadm module correctly? Thanks in advance for any help!

2 2

Misplaced objects greater than 100%
by Johan Hattne 06 Apr '23

06 Apr '23

Dear all; Up until a few hours ago, I had a seemingly normally-behaving cluster (Quincy, 17.2.5) with 36 OSDs, evenly distributed across 3 of its 6 nodes. The cluster is only used for CephFS and the only non-standard configuration I can think of is that I had 2 active MDSs, but only 1 standby. I had also doubled mds_cache_memory limit to 8 GB (all OSD hosts have 256 G of RAM) at some point in the past. Then I rebooted one of the OSD nodes. The rebooted node held one of the active MDSs. Now the node is back up: ceph -s says the cluster is healthy, but all PGs are in a active+clean+remapped state and 166.67% of the objects are misplaced (dashboard: -66.66% healthy). The data pool is a threefold replica with 5.4M object, the number of misplaced objects is reported as 27087410/16252446. The denominator in the ratio makes sense to me (16.2M / 3 = 5.4M), but the numerator does not. I also note that the ratio is *exactly* 5 / 3. The filesystem is still mounted and appears to be usable, but df reports it as 100% full; I suspect it would say 167% but that is capped somewhere. Any ideas about what is going on? Any suggestions for recovery? // Best wishes; Johan

3 6

CephFS thrashing through the page cache
by Ashu Pachauri 05 Apr '23

05 Apr '23

We have an internal use case where we back the storage of a proprietary database by a shared file system. We noticed something very odd when testing some workload with a local block device backed file system vs cephfs. We noticed that the amount of network IO done by cephfs is almost double compared to the IO done in case of a local file system backed by an attached block device. We also noticed that CephFS thrashes through the page cache very quickly compared to the amount of data being read and think that the two issues might be related. So, I wrote a simple test. 1. I wrote 10k files 400KB each using dd (approx 4 GB data). 2. I dropped the page cache completely. 3. I then read these files serially, again using dd. The page cache usage shot up to 39 GB for reading such a small amount of data. Following is the code used to repro this in bash: for i in $(seq 1 10000); do dd if=/dev/zero of=test_${i} bs=4k count=100 done sync; echo 1 > /proc/sys/vm/drop_caches for i in $(seq 1 10000); do dd if=test_${i} of=/dev/null bs=4k count=100 done The ceph version being used is: ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable) The ceph configs being overriden: WHO MASK LEVEL OPTION VALUE RO mon advanced auth_allow_insecure_global_id_reclaim false mgr advanced mgr/balancer/mode upmap mgr advanced mgr/dashboard/server_addr 127.0.0.1 * mgr advanced mgr/dashboard/server_port 8443 * mgr advanced mgr/dashboard/ssl false * mgr advanced mgr/prometheus/server_addr 0.0.0.0 * mgr advanced mgr/prometheus/server_port 9283 * osd advanced bluestore_compression_algorithm lz4 osd advanced bluestore_compression_mode aggressive osd advanced bluestore_throttle_bytes 536870912 osd advanced osd_max_backfills 3 osd advanced osd_op_num_threads_per_shard_ssd 8 * osd advanced osd_scrub_auto_repair true mds advanced client_oc false mds advanced client_readahead_max_bytes 4096 mds advanced client_readahead_max_periods 1 mds advanced client_readahead_min 0 mds basic mds_cache_memory_limit 21474836480 client advanced client_oc false client advanced client_readahead_max_bytes 4096 client advanced client_readahead_max_periods 1 client advanced client_readahead_min 0 client advanced fuse_disable_pagecache false The cephfs mount options (note that readahead was disabled for this test): /mnt/cephfs type ceph (rw,relatime,name=cephfs,secret=<hidden>,acl,rasize=0) Any help or pointers are appreciated; this is a major performance issue for us. Thanks and Regards, Ashu Pachauri

4 12

Upgrading to 16.2.11 timing out on ceph-volume due to raw list performance bug, downgrade isn't possible due to new OP code in bluestore
by Mikael Öhman 05 Apr '23

05 Apr '23

Trying to upgrade a containerized setup from 16.2.10 to 16.2.11 gave us two big surprises, I wanted to share in case anyone else encounters the same. I don't see any nice solution to this apart from a new release that fixes the performance regression that completely breaks the container setup in cephadm due to timeouts: After some digging, we would that the it was the "ceph-volume" command that kept timing out, and after a ton of digging, found that it does so because of https://github.com/ceph/ceph/commit/bea9f4b643ce32268ad79c0fc257b25ff2f8333… which was introduced into 16.2.11. Unfortunately, the vital fix for this https://github.com/ceph/ceph/commit/8d7423c3e75afbe111c91e699ef3cb1c0beee61b was not included in 16.2.11 So, in a setup like ours, with *many* devices, a simple "ceph-volume raw list" now takes over 10 minutes to run (instead of 5 seconds in 16.2.10). As a result, the service files that cephadm generates [Service] LimitNOFILE=1048576 LimitNPROC=1048576 EnvironmentFile=-/etc/environment ExecStart=/bin/bash /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/%i/unit.run ExecStop=-/bin/bash -c '/bin/podman stop ceph-5406fed0-d52b-11ec-beff-7ed30a54847b-%i ; bash /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/%i/unit.stop' ExecStopPost=-/bin/bash /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/%i/unit.poststop KillMode=none Restart=on-failure RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=120 StartLimitInterval=30min StartLimitBurst=5 ExecStartPre=-/bin/rm -f %t/%n-pid %t/%n-cid ExecStopPost=-/bin/rm -f %t/%n-pid %t/%n-cid Type=forking PIDFile=%t/%n-pid Delegate=yes will repeatedly be marked as failed, as they take over 2 minutes to run now. This tells systemd to restart, and we now have an infinite loop, as the 5 restarts takes over 50 minutes, it never even triggers the StarLimitInterval, leaving this OSD in an infinite loop over listing the n^2 devices (which, as a bonus, is also filling up the root disk with an enormous amount of repeated logging in ceph-volume.log as it infinitely tries to figure out if a block device is a bluestore) And trying to just fix the service or unit files manually to at least just stop this container from being incorrectly restarted over and over, is also a dead end, since the orchestration stuff just overwrites this automatically, and restarts the services again. I found it seemed to be /var/lib/ceph/5406fed0-d52b-11ec-beff-7ed30a54847b/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2 on my system that generated these files, so i tried tweaking that to have the necessary 1200 second TimeoutStart and finally that managed to get the darn container to stop restarting endlessly. (I admit i'm very fuzzy on how these services and orchestration stuff is triggered as i usually don't work on our storage stuff) Still though, it takes 11 minutes to start each OSD service now, so this isn't great. We wanted to revert back to 16.2.10 but it turns out to also be a no-go, as a new operation added to bluefs https://github.com/ceph/ceph/pull/42750 in 16.2.11 (though this isn't mentioned in the changelogs, i had to compare the source code to see that it was in fact added 16.2.11). So trying to revert an OSD then fails with: debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluefs _replay 0x100000: stop: unrecognized op 12 debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluefs mount failed to replay log: (5) Input/output error debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluestore(/var/lib/ceph/osd/ceph-10) _open_bluefs failed bluefs mount: (5) Input/output error debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 -1 bluestore(/var/lib/ceph/osd/ceph-10) _open_db failed to prepare db environment: debug 2023-04-04T11:42:45.927+0000 7f2c12f6a200 1 bdev(0x5590e80a0400 /var/lib/ceph/osd/ceph-10/block) close debug 2023-04-04T11:42:46.153+0000 7f2c12f6a200 -1 osd.10 0 OSD:init: unable to mount object store debug 2023-04-04T11:42:46.153+0000 7f2c12f6a200 -1 ** ERROR: osd init failed: (5) Input/output error Ouch Best regards, Mikael

3 3

RGW can't create bucket
by Kamil Madac 05 Apr '23

05 Apr '23

Hi, One of my customers had a correctly working RGW cluster with two zones in one zonegroup and since a few days ago users are not able to create buckets and are always getting Access denied. Working with existing buckets works (like listing/putting objects into existing bucket). The only operation which is not working is bucket creation. We also tried to create a new user, but the behavior is the same, and he is not able to create the bucket. We tried s3cmd, python script with boto library and also Dashboard as admin user. We are always getting Access Denied. Zones are in-sync. Has anyone experienced such behavior? Thanks in advance, here are some outputs: $ s3cmd -c .s3cfg_python_client mb s3://test ERROR: Access to bucket 'test' was denied ERROR: S3 error: 403 (AccessDenied) Zones are in-sync: Primary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) metadata sync no sync (zone is master) data sync source: e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source Secondary cluster: # radosgw-admin sync status realm 5429b434-6d43-4a18-8f19-a5720a89c621 (solargis-prod) zonegroup 00e4b3ff-1da8-4a86-9f52-4300c6d0f149 (solargis-prod-ba) zone e84fd242-dbae-466c-b4d9-545990590995 (solargis-prod-ba-hq) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6067eec6-a930-45c7-af7d-a7ef2785a2d7 (solargis-prod-ba-dc) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source -- Kamil Madac

3 6

Crushmap rule for multi-datacenter erasure coding
by Michel Jouvin 05 Apr '23

05 Apr '23

Hi, We have a 3-site Ceph cluster and would like to create a 4+2 EC pool with 2 chunks per datacenter, to maximise the resilience in case of 1 datacenter being down. I have not found a way to create an EC profile with this 2-level allocation strategy. I created an EC profile with a failure domain = datacenter but it doesn't work as, I guess, it would like to ensure it has always 5 OSDs up (to ensure that the pools remains R/W) where with a failure domain = datacenter, the guarantee is only 4. My idea was to create a 2-step allocation and a failure domain=host to achieve our desired configuration, with something like the following in the crushmap rule: step choose indep 3 datacenter step chooseleaf indep x host step emit Is it the right approach? If yes, what should be 'x'? Would 0 work? From what I have seen, there is no way to create such a rule with the 'ceph osd crush' commands: I have to download the current CRUSHMAP, edit it and upload the modified version. Am I right? Thanks in advance for your help or suggestions. Best regards, Michel

2 1

Crushmap rule for multi-datacenter erasure coding
by Michel Jouvin 04 Apr '23

04 Apr '23

Hi, We have a 3-site Ceph cluster and would like to create a 4+2 EC pool with 2 chunks per datacenter, to maximise the resilience in case of 1 datacenter being down. I have not found a way to create an EC profile with this 2-level allocation strategy. I created an EC profile with a failure domain = datacenter but it doesn't work as, I guess, it would like to ensure it has always 5 OSDs up (to ensure that the pools remains R/W) where with a failure domain = datacenter, the guarantee is only 4. My idea was to create a 2-step allocation and a failure domain=host to achieve our desired configuration, with something like the following in the crushmap rule: step choose indep 3 datacenter step chooseleaf indep x host step emit Is it the right approach? If yes, what should be 'x'? Would 0 work? From what I have seen, there is no way to create such a rule with the 'ceph osd crush' commands: I have to download the current CRUSHMAP, edit it and upload the modified version. Am I right? Thanks in advance for your help or suggestions. Best regards, Michel

3 5

Recently deployed cluster showing 9Tb of raw usage without any load deployed
by Work Ceph 04 Apr '23

04 Apr '23

Hello guys! We noticed an unexpected situation. In a recently deployed Ceph cluster we are seeing a raw usage, that is a bit odd. We have the following setup: We have a new cluster with 5 nodes with the following setup: - 128 GB of RAM - 2 cpus Intel(R) Intel Xeon Silver 4210R - 1 NVME of 2 TB for the rocks DB caching - 5 HDDs of 14TB - 1 NIC dual port of 25GiB in BOND mode. Right after deploying the Ceph cluster, we see a raw usage of about 9TiB. However, no load has been applied onto the cluster. Have you guys seen such a situation? Or, can you guys help understand it? We are using Ceph Octopus, and we have set the following configurations: ``` ceph_conf_overrides: global: osd pool default size: 3 osd pool default min size: 1 osd pool default pg autoscale mode: "warn" perf: true rocksdb perf: true mon: mon osd down out interval: 120 osd: bluestore min alloc size hdd: 65536 ``` Any tip or help on how to explain this situation is welcome!

3 7

Read and write performance on distributed filesystem
by David Cunningham 04 Apr '23

04 Apr '23

Hello, We are considering CephFS as an alternative to GlusterFS, and have some questions about performance. Is anyone able to advise us please? This would be for file systems between 100GB and 2TB in size, average file size around 5MB, and a mixture of reads and writes. I may not be using the correct terminology in the Ceph world, but in my parlance a node is a Linux server running the Ceph storage software. Multiple nodes make up the whole Ceph storage solution. Someone correct me if I should be using different terms! In our normal scenario the nodes in the replicated filesystem would be around 0.3ms apart, but we're also interested in geographically remote nodes which would be say 20ms away. We are using third party software which relies on a traditional Linux filesystem, so we can't use an object storage solution directly. So my specific questions are: 1. When reading a file from CephFS, does it read from just one node, or from all nodes? 2. If reads are from one node then does it choose the node with the fastest response to optimise performance, or if from all nodes then will reads be no faster than latency to the furthest node? 3. When writing to CephFS, are all nodes written to synchronously, or are writes to one node which then replicates that to other nodes asynchronously? 4. Can anyone give a recommendation on maximum latency between nodes to have decent performance? 5. How does CephFS handle a node which suddenly becomes unavailable on the network? Is the block time configurable, and how good is the healing process after the lost node rejoins the network? 6. I have read that CephFS is more complicated to administer than GlusterFS. What does everyone think? Are things like healing after a net split difficult for administrators new to Ceph to handle? Thanks very much in advance. -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782

2 1

compiling Nautilus for el9
by Marc 03 Apr '23

03 Apr '23

Is it possible to compile Nautilus for el9? Or maybe just the osd's?

2 4

How mClock profile calculation works, and IOPS
by Luis Domingues 03 Apr '23

03 Apr '23

Hi, I am reading reading some documentation about mClock and have two questions. First, about the IOPS. Are those IOPS disk IOPS or other kind of IOPS? And what the assumption of those? (Like block size, sequential or random reads/writes)? And the second question, How mClock calculates its profiles? I have my lab cluster running Quincy, and I have this parameters for mClock: "osd_mclock_max_capacity_iops_hdd": "450.000000", "osd_mclock_profile": "balanced", According to the documentation: https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/#bala… I am expecting to have: "osd_mclock_scheduler_background_best_effort_lim": "999999", "osd_mclock_scheduler_background_best_effort_res": "90", "osd_mclock_scheduler_background_best_effort_wgt": "2", "osd_mclock_scheduler_background_recovery_lim": "675", "osd_mclock_scheduler_background_recovery_res": "180", "osd_mclock_scheduler_background_recovery_wgt": "1", "osd_mclock_scheduler_client_lim": "450", "osd_mclock_scheduler_client_res": "180", "osd_mclock_scheduler_client_wgt": "1", But what I get is: "osd_mclock_scheduler_background_best_effort_lim": "999999", "osd_mclock_scheduler_background_best_effort_res": "18", "osd_mclock_scheduler_background_best_effort_wgt": "2", "osd_mclock_scheduler_background_recovery_lim": "135", "osd_mclock_scheduler_background_recovery_res": "36", "osd_mclock_scheduler_background_recovery_wgt": "1", "osd_mclock_scheduler_client_lim": "90", "osd_mclock_scheduler_client_res": "36", "osd_mclock_scheduler_client_wgt": "1", Which seems very low according to what my disk seems to be able to handle. Is this calculation the expected one? Or did I miss something on how those profiles are populated? Luis Domingues Proton AG

2 5

Eccessive occupation of small OSDs
by Nicola Mori 03 Apr '23

03 Apr '23

Dear Ceph users, my cluster is made up of 10 old machines, with uneven number of disks and disk size. Essentially I have just one big data pool (6+2 erasure code, with host failure domain) for which I am currently experiencing a very poor available space (88 TB of which 40 TB occupied, as reported by df -h on hosts mounting the cephfs) compared to the raw one (196.5 TB). I have a total of 104 OSDs and 512 PGs for the pool; I cannot increment the PG number since the machines are old and with very low amount of RAM, and some of them are already overloaded. In this situation I'm seeing a high occupation of small OSDs (500 MB) with respect to bigger ones (2 and 4 TB) even if the weight is set equal to disk capacity (see below for ceph osd tree). For example OSD 9 is at 62% occupancy even with weight 0.5 and reweight 0.75, while the highest occupancy for 2 TB OSDs is 41% (OSD 18) and 4 TB OSDs is 23% (OSD 79). I guess this high occupancy for 500 MB OSDs combined with erasure code size and host failure domain might be the cause of the poor available space, could this be true? The upmap balancer is currently running but I don't know if and how much it could improve the situation. Any hint is greatly appreciated, thanks. Nicola # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 196.47754 root default -7 14.55518 host aka 4 hdd 1.81940 osd.4 up 1.00000 1.00000 11 hdd 1.81940 osd.11 up 1.00000 1.00000 18 hdd 1.81940 osd.18 up 1.00000 1.00000 26 hdd 1.81940 osd.26 up 1.00000 1.00000 32 hdd 1.81940 osd.32 up 1.00000 1.00000 41 hdd 1.81940 osd.41 up 1.00000 1.00000 48 hdd 1.81940 osd.48 up 1.00000 1.00000 55 hdd 1.81940 osd.55 up 1.00000 1.00000 -3 14.55518 host balin 0 hdd 1.81940 osd.0 up 1.00000 1.00000 8 hdd 1.81940 osd.8 up 1.00000 1.00000 15 hdd 1.81940 osd.15 up 1.00000 1.00000 22 hdd 1.81940 osd.22 up 1.00000 1.00000 29 hdd 1.81940 osd.29 up 1.00000 1.00000 34 hdd 1.81940 osd.34 up 1.00000 1.00000 43 hdd 1.81940 osd.43 up 1.00000 1.00000 49 hdd 1.81940 osd.49 up 1.00000 1.00000 -13 29.10950 host bifur 3 hdd 3.63869 osd.3 up 1.00000 1.00000 14 hdd 3.63869 osd.14 up 1.00000 1.00000 27 hdd 3.63869 osd.27 up 1.00000 1.00000 37 hdd 3.63869 osd.37 up 1.00000 1.00000 50 hdd 3.63869 osd.50 up 1.00000 1.00000 59 hdd 3.63869 osd.59 up 1.00000 1.00000 64 hdd 3.63869 osd.64 up 1.00000 1.00000 69 hdd 3.63869 osd.69 up 1.00000 1.00000 -17 29.10950 host bofur 2 hdd 3.63869 osd.2 up 1.00000 1.00000 21 hdd 3.63869 osd.21 up 1.00000 1.00000 39 hdd 3.63869 osd.39 up 1.00000 1.00000 57 hdd 3.63869 osd.57 up 1.00000 1.00000 66 hdd 3.63869 osd.66 up 1.00000 1.00000 72 hdd 3.63869 osd.72 up 1.00000 1.00000 76 hdd 3.63869 osd.76 up 1.00000 1.00000 79 hdd 3.63869 osd.79 up 1.00000 1.00000 -21 29.10376 host dwalin 88 hdd 1.81898 osd.88 up 1.00000 1.00000 89 hdd 1.81898 osd.89 up 1.00000 1.00000 90 hdd 1.81898 osd.90 up 1.00000 1.00000 91 hdd 1.81898 osd.91 up 1.00000 1.00000 92 hdd 1.81898 osd.92 up 1.00000 1.00000 93 hdd 1.81898 osd.93 up 1.00000 1.00000 94 hdd 1.81898 osd.94 up 1.00000 1.00000 95 hdd 1.81898 osd.95 up 1.00000 1.00000 96 hdd 1.81898 osd.96 up 1.00000 1.00000 97 hdd 1.81898 osd.97 up 1.00000 1.00000 98 hdd 1.81898 osd.98 up 1.00000 1.00000 99 hdd 1.81898 osd.99 up 1.00000 1.00000 100 hdd 1.81898 osd.100 up 1.00000 1.00000 101 hdd 1.81898 osd.101 up 1.00000 1.00000 102 hdd 1.81898 osd.102 up 1.00000 1.00000 103 hdd 1.81898 osd.103 up 1.00000 1.00000 -9 14.55518 host ogion 7 hdd 1.81940 osd.7 up 1.00000 1.00000 16 hdd 1.81940 osd.16 up 1.00000 1.00000 23 hdd 1.81940 osd.23 up 1.00000 1.00000 33 hdd 1.81940 osd.33 up 1.00000 1.00000 40 hdd 1.81940 osd.40 up 1.00000 1.00000 47 hdd 1.81940 osd.47 up 1.00000 1.00000 54 hdd 1.81940 osd.54 up 1.00000 1.00000 61 hdd 1.81940 osd.61 up 1.00000 1.00000 -19 14.55518 host prestno 81 hdd 1.81940 osd.81 up 1.00000 1.00000 82 hdd 1.81940 osd.82 up 1.00000 1.00000 83 hdd 1.81940 osd.83 up 1.00000 1.00000 84 hdd 1.81940 osd.84 up 1.00000 1.00000 85 hdd 1.81940 osd.85 up 1.00000 1.00000 86 hdd 1.81940 osd.86 up 1.00000 1.00000 87 hdd 1.81940 osd.87 up 1.00000 1.00000 104 hdd 1.81940 osd.104 up 1.00000 1.00000 -15 29.10376 host remolo 6 hdd 1.81897 osd.6 up 1.00000 1.00000 12 hdd 1.81897 osd.12 up 1.00000 1.00000 19 hdd 1.81897 osd.19 up 1.00000 1.00000 28 hdd 1.81897 osd.28 up 1.00000 1.00000 35 hdd 1.81897 osd.35 up 1.00000 1.00000 44 hdd 1.81897 osd.44 up 1.00000 1.00000 52 hdd 1.81897 osd.52 up 1.00000 1.00000 58 hdd 1.81897 osd.58 up 1.00000 1.00000 63 hdd 1.81897 osd.63 up 1.00000 1.00000 67 hdd 1.81897 osd.67 up 1.00000 1.00000 71 hdd 1.81897 osd.71 up 1.00000 1.00000 73 hdd 1.81897 osd.73 up 1.00000 1.00000 74 hdd 1.81897 osd.74 up 1.00000 1.00000 75 hdd 1.81897 osd.75 up 1.00000 1.00000 77 hdd 1.81897 osd.77 up 1.00000 1.00000 78 hdd 1.81897 osd.78 up 1.00000 1.00000 -5 14.55518 host rokanan 1 hdd 1.81940 osd.1 up 1.00000 1.00000 10 hdd 1.81940 osd.10 up 1.00000 1.00000 17 hdd 1.81940 osd.17 up 1.00000 1.00000 24 hdd 1.81940 osd.24 up 1.00000 1.00000 31 hdd 1.81940 osd.31 up 1.00000 1.00000 38 hdd 1.81940 osd.38 up 1.00000 1.00000 46 hdd 1.81940 osd.46 up 1.00000 1.00000 53 hdd 1.81940 osd.53 up 1.00000 1.00000 -11 7.27515 host romolo 5 hdd 0.45470 osd.5 up 1.00000 1.00000 9 hdd 0.45470 osd.9 up 0.75000 1.00000 13 hdd 0.45470 osd.13 up 1.00000 1.00000 20 hdd 0.45470 osd.20 up 0.95000 1.00000 25 hdd 0.45470 osd.25 up 0.75000 1.00000 30 hdd 0.45470 osd.30 up 1.00000 1.00000 36 hdd 0.45470 osd.36 up 1.00000 1.00000 42 hdd 0.45470 osd.42 up 1.00000 1.00000 45 hdd 0.45470 osd.45 up 0.85004 1.00000 51 hdd 0.45470 osd.51 up 0.89999 1.00000 56 hdd 0.45470 osd.56 up 1.00000 1.00000 60 hdd 0.45470 osd.60 up 1.00000 1.00000 62 hdd 0.45470 osd.62 up 1.00000 1.00000 65 hdd 0.45470 osd.65 up 0.85004 1.00000 68 hdd 0.45470 osd.68 up 1.00000 1.00000 70 hdd 0.45470 osd.70 up 1.00000 1.00000

4 5

Re: monitoring apply_latency / commit_latency ?
by Konstantin Shalygin 03 Apr '23

03 Apr '23

Hi, > On 2 Apr 2023, at 23:14, Matthias Ferdinand <mf+ml.ceph(a)mfedv.net> wrote: > > I understand that grafana graphs are generated from prometheus metrics. > I just wanted to know which OSD daemon-perf values feed these prometheus > metrics (or if they are generated in some other way). Yep, this perf metrics is generated in some way 🙂 You can consult with ceph-mgr prometheus module source code [1] [1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/prometheus/module.py#… k

1 0

Set the Quality of Service configuration.
by farhad kh 02 Apr '23

02 Apr '23

how to i can set IO quota or limit for R/W for erasure coding pool in ceph ?

3 2

monitoring apply_latency / commit_latency ?
by Matthias Ferdinand 02 Apr '23

02 Apr '23

Hi, I would like to understand how the per-OSD data from "ceph osd perf" (i.e. apply_latency, commit_latency) is generated. So far I couldn't find documentation on this. "ceph osd perf" output is nice for a quick glimpse, but is not very well suited for graphing. Output values are from the most recent 5s-averages apparently. With "ceph daemon osd.X perf dump" OTOH, you get quite a lot of latency metrics, while it is just not obvious to me how they aggregate into apply_latency and commit_latency. Or some comparably easy read latency metric (something that is missing completely in "ceph osd perf"). Can somebody shed some light on this? Regards Matthias

2 4

avg apply latency went up after update from octopus to pacific
by Boris Behrens 02 Apr '23

02 Apr '23

Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms. All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value? Cheers Boris

10 27

Failing to create monitor in a working cluster.
by pmestre＠gmail.com 01 Apr '23

01 Apr '23

Hello, i've been running a 3 node proxmox cluster with 4 ceph osd for 3 years as a production cluster. As a test for trying to move ceph cluster network, i destroyed one of the 3 working monitors and tried to recreate it. After destroying it, the new monitor refuses to join the cluster, even in the old network. I've tried all steps in documentation "Troubleshooting monitors" section. New monitor has this config extracted from ceph --admin-daemon file.asok { "name": "n3ceph", "rank": -1, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "2449958197560098820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "0", "quorum_mon": [] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false } The quorum mon stat is as follows: { "name": "node1", "rank": 0, "state": "leader", "election_epoch": 340, "quorum": [ 0, 1 ], "quorum_age": 13090, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138314316775423", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 5 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 1 }, { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 12 } ], "mgr": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false I tried to get a debug log with ceph daemon mon.n3ceph config set debug_mon 10/10 and restarting the service, but the ceph log file stoped working after i tried that setting. journalctl -u tells me: mar 31 17:35:22 node3 ceph-mon[240916]: 2023-03-31T17:35:22.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:27 node3 ceph-mon[240916]: 2023-03-31T17:35:27.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:32 node3 ceph-mon[240916]: 2023-03-31T17:35:32.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200). Any ideas? Cluster is running fine with two monitors, but a reboot in one of the nodes might be a big problem. Kind regards and many thanks.

2 1