- ceph-users - lists.ceph.io

Unhealthy Cluster | Remove / Purge duplicate osds | Fix daemon
by Oliver Weinmann 16 Mar '21

16 Mar '21

Hi, On my 3 node Octopus 15.2.5 test cluster, that I haven't used for quite a while, I noticed that it shows some errors: [root@gedasvl02 ~]# ceph health detail INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 HEALTH_WARN 2 failed cephadm daemon(s) [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) daemon osd.0 on gedaopl02 is in error state daemon node-exporter.gedaopl01 on gedaopl01 is in error state The error about the osd.0 is strange since osd.0 is actually up and running but on a different node. I guess I missed to correctly remove it from node gedaopl02 and then added a new osd to a different node gedaopl01 and now there are duplicate osd ids for osd.0 and osd.2. [root@gedasvl02 ~]# ceph orch ps INFO:cephadm:Inferring fsid d0920c36-2368-11eb-a5de-005056b703af INFO:cephadm:Inferring config /var/lib/ceph/d0920c36-2368-11eb-a5de-005056b703af/mon.gedasvl02/config INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID alertmanager.gedasvl02 gedasvl02 running (6h) 7m ago 4M 0.20.0 docker.io/prom/alertmanager:v0.20.0 0881eb8f169f 5b80fb977a5f crash.gedaopl01 gedaopl01 stopped 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 810cf432b6d6 crash.gedaopl02 gedaopl02 running (5h) 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 34ab264fd5ed crash.gedaopl03 gedaopl03 running (2d) 7m ago 2d 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 233f30086d2d crash.gedasvl02 gedasvl02 running (6h) 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 ea3d3e7c4f58 grafana.gedasvl02 gedasvl02 running (6h) 7m ago 4M 6.6.2 docker.io/ceph/ceph-grafana:6.6.2 a0dce381714a 5a94f3e41c32 mds.cephfs.gedaopl01.zjuhem gedaopl01 stopped 7m ago 3M <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mds.cephfs.gedasvl02.xsjtpi gedasvl02 running (6h) 7m ago 3M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 26e7c8759d89 mgr.gedaopl03.zilwbl gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 e18b6f40871c mon.gedaopl03 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 5afdf40e41ba mon.gedasvl02 gedasvl02 running (6h) 7m ago 4M 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 e83dfcd864aa node-exporter.gedaopl01 gedaopl01 error 7m ago 4M 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 0fefcfcc9639 node-exporter.gedaopl02 gedaopl02 running (5h) 7m ago 4M 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf f459045b7e41 node-exporter.gedaopl03 gedaopl03 running (2d) 7m ago 2d 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 3bd9f8dd6d5b node-exporter.gedasvl02 gedasvl02 running (6h) 7m ago 4M 0.18.1 docker.io/prom/node-exporter:v0.18.1 e5a616e4b9cf 72e96963261e *osd.0 gedaopl01 running (5h) 7m ago 5h 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 ed76fafb1988** **osd.0 gedaopl02 error 7m ago 4M <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown>* osd.1 gedaopl01 running (4h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 41a43733e601 *osd.2 gedaopl01 stopped 7m ago 4M <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown>** **osd.2 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 ac9e660db2fb* osd.3 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 bde17b5bb2fb osd.4 gedaopl02 running (5h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 7cc3ef7c4469 osd.5 gedaopl02 running (5h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 761b96d235e4 osd.6 gedaopl02 running (5h) 7m ago 3d 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 d047b28fe2bd osd.7 gedaopl03 running (7h) 7m ago 7h 15.2.9 docker.io/ceph/ceph:v15 dfc483079636 3b54b01841f4 osd.8 gedaopl01 running (5h) 7m ago 5h 15.2.5 docker.io/ceph/ceph:v15 4405f6339e35 cdd308cdc82b prometheus.gedasvl02 gedasvl02 running (5h) 7m ago 4M 2.18.1 docker.io/prom/prometheus:v2.18.1 de242295e225 591cef3bbaa4 Is there a way to clean / purge the stopped and error ones? I don't know what is wrong with the node-exporter. Because looking at podman ps -a on gedaopl01 looks ok. Maybe also a zombie daemon? [root@gedaopl01 ~]# podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e71898f7d038 docker.io/prom/node-exporter:v0.18.1 --no-collector.ti... 54 seconds ago Up 54 seconds ago ceph-d0920c36-2368-11eb-a5de-005056b703af-node-exporter.gedaopl01 41a43733e601 docker.io/ceph/ceph:v15 -n osd.1 -f --set... 5 hours ago Up 5 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.1 810cf432b6d6 docker.io/ceph/ceph:v15 -n client.crash.g... 6 hours ago Up 6 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-crash.gedaopl01 cdd308cdc82b docker.io/ceph/ceph:v15 -n osd.8 -f --set... 6 hours ago Up 6 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.8 ed76fafb1988 docker.io/ceph/ceph:v15 -n osd.0 -f --set... 6 hours ago Up 6 hours ago ceph-d0920c36-2368-11eb-a5de-005056b703af-osd.0 I replaced the very old disks with some brand new SAMSUMG PM883 and would like to upgrade to 15.2.9. But the upgrade guide recommends to do this on a healthy cluster only. :) Cheers, Oliver

2 3

Safe to remove osd or not? Which statement is correct?
by Szabo, Istvan (Agoda) 15 Mar '21

15 Mar '21

Hi Gents, There is a cluster with 14 hosts in this state: https://i.ibb.co/HPF3Pdr/6-ACB2-C5-B-6-B54-476-B-835-D-227-E9-BFB1247.jpg There is a host based crush rule ec 3:1 and there are 3 hosts where are osds down. Unfortunately there are pools with 3 replicas also which is host based. 2 hosts have 2 osds down,1 host has 1 osd down, which means I guess if we don’t bring it back, 0.441% data loss is going to happen. Am I right? I hope I’m not right because there isn’t any missplaced objects, only degraded. The problem is that the osds are not starting so somehow these 5 osds should be removed, but I’m curious which is my statement is correct. Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

millions slow ops on a cluster without load
by Szabo, Istvan (Agoda) 15 Mar '21

15 Mar '21

We have a cluster with a huge amount of warnings like this even if nothing is going on in the cluster. It makes mgr physical memory full, mon db maxed out 5 osds can't start :/ [WRN] slow request osd_op(mds.0.537792:26453 43.38 43:1d6c5587:::10000fe56a6.00000000:head [create,setxattr parent (367) in=373b,setxattr layout (30) in=36b] snapc 0=[] RETRY=131 ondisk+retry+write+known_if_redirected+full_force e5532574) initiated 2021-03-15T16:44:18.428585+0100 currently delayed Is there a way to somehow stop it? MDS service stopped also. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

v14.2.18 Nautilus released
by David Galloway 15 Mar '21

15 Mar '21

We're happy to announce the 18th backport release in the Nautilus series. It fixes a regression introduced in 14.2.17 in which the manager module tries to use a couple python modules that do not exist in some environments. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-18-nautilus-released Notable Changes --------------- * This release fixes issues loading the dashboard and volumes manager modules in some environments. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.18.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: befbc92f3c11eedd8626487211d200c0b44786d9

1 0

Best way to add OSDs - whole node or one by one?
by Dave Hall 15 Mar '21

15 Mar '21

Hello, I am currently in the process of expanding my Nautilus cluster from 3 nodes (combined OSD/MGR/MON/MDS) to 6 OSD nodes and 3 management nodes. The old and new OSD nodes all have 8 x 12TB HDDs plus NVMe. The front and back networks are 10GB. Last Friday evening I injected a whole new OSD node, increasing the OSD HDDs from 24 to 32. As of this morning the cluster is still re-balancing - with periodic warnings about degraded PGs and missed deep-scrub deadlines. So after 4.5 days my misplaced PGs are down from 33% to 2%. My question: For a cluster of this size, what is the best-practice procedure for adding OSDs? Should I use 'ceph-volume prepare' to layout the new OSDs, but only add them a couple at a time, or should I continue adding whole nodes? Maybe this has to do with a maximum percentage of misplaced PGs. The first new node increased the OSD capacity by 33% and resulted in 33% PG misplacement. The next node will only result in 25% misplacement. If a too high percentage of misplaced PGs negatively impacts rebalancing or data availability, what is a reasonable ceiling for this percentage? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office)

4 5

MDS pinning: ceph.dir.pin: No such attribute
by Frank Schilder 15 Mar '21

15 Mar '21

Hi, I seem to have a problem with the extended attributes for MDS pinning. Ceph version is mimic-13.2.10 and the documentation (https://web.archive.org/web/20190716110503/http://docs.ceph.com/docs/mimic/…) says I should be able to setfattr -n ceph.dir.pin -v 2 path/to/dir When I try this, I get the following: # setfattr -n ceph.dir.pin -v 1 . # getfattr -n ceph.dir.pin . .: ceph.dir.pin: No such attribute Is this expected? How can I check that pinning is in effect as intended? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 5

Current BlueStore cache autotune (memory target) is respect media?
by Konstantin Shalygin 15 Mar '21

15 Mar '21

Hi, Current default's (for nautilus, for example) is respect media type (hdd/ssd/hybrid) or for all OSD's current memory_target is 4GiB? Thanks, k

2 2

lvm fix for reseated reseated device
by Dan van der Ster 15 Mar '21

15 Mar '21

Hi all, Occasionally we see a bus glitch which causes a device to disappear then reappear with a new /dev/sd name. This crashes the osd (giving IO errors) but after a reboot the OSD will be perfectly fine. We're looking for a way to reeactivate osd like this without rebooting. For example, logs showing sdd disappear then reappear as sdq from this morning are in the P.S. We tried pvscan, vgscan, lvscan, but in all cases when trying to activate the osd we get an I/O error, as if the dm entry for the lv/vg is still referring to /dev/sdd. Is there some obvious way to properly tear down what refers to sdd / 0:0:3:0 so that we can activate sdq ? (In this case, we have already rebooted the box so I won't be able to test immediately.) Best Regards, Dan Mar 15 04:57:36 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: device_block, handle(0x001c) Mar 15 04:57:38 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: device_unblock and setting to running, handle(0x001c) Mar 15 04:57:38 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: [sdd] Synchronizing SCSI cache Mar 15 04:57:38 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:3:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Stopping LVM event activation on device 8:48... Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x300062b2038af0c3) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: removing handle(0x001c), sas_addr(0x300062b2038af0c3) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: enclosure logical id(0x500062b2038af0c0), slot(1) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch lvm[1157119]: pvscan[1157119] device 8:48 not found. Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: lvm2-pvscan@8:48.service: Succeeded. Mar 15 04:57:39 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Stopped LVM event activation on device 8:48. ... Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: Direct-Access ATA Micron_5200_MTFD U020 PQ: 0 ANSI: 6 Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: SATA: handle(0x001c), sas_addr(0x300062b2038af0c3), phy(3), device_name(0x0000000000000000) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: enclosure logical id (0x500062b2038af0c0), slot(1) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: enclosure level(0x0000), connector name( ) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: scsi 0:0:16:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: Power-on or device reset occurred Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: Attached scsi generic sg3 type 0 Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: end_device-0:16: add: handle(0x001c), sas_addr(0x300062b2038af0c3) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] 1875385008 512-byte logical blocks: (960 GB/894 GiB) Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] 4096-byte physical blocks Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] Write Protect is off Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch kernel: sd 0:0:16:0: [sdq] Attached SCSI disk Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Starting LVM event activation on device 65:0... Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: pvscan[1157327] PV /dev/sdq online, VG ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5 is complete. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: pvscan[1157327] VG ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5 run autoactivation. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: PVID qlETw1-KssL-Vc9P-MpAE-ED7I-gnWS-kO4DWF read from /dev/sdq last written to /dev/sdd. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: pvscan[1157327] VG ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5 not using quick activation. Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch lvm[1157327]: 1 logical volume(s) in volume group "ceph-2b92ed55-2e7a-4aba-aab5-899b071eceb5" now active Mar 15 04:58:16 cephflash21b-b3b91f0bb3.cern.ch systemd[1]: Started LVM event activation on device 65:0.

3 3

Alertmanager not using custom configuration template
by Marc 'risson' Schmitt 15 Mar '21

15 Mar '21

Hi, I'm trying to use a custom template for Alertmanager deployed with Cephadm. Following its documentation[1], I set the option `mgr/cephadm/alertmanager_alertmanager.yml` to my own template, restarted the mgr, and re-deployed Alertmanager. However, Cephadm seems to always use its internal template. After some debugging, I found that the mgr indeed queries for `mgr/cephadm/alertmanager_alertmanager.yml`, but the end template always is the default one. Am I doing something wrong? Is anyone else having this issue? [1] https://docs.ceph.com/en/octopus/cephadm/monitoring/#using-custom-configura… Regards, -- Marc 'risson' Schmitt CRI - EPITA

2 4

MDS stuck in replay/resolve stats
by Zhenshi Zhou 15 Mar '21

15 Mar '21

Hi, I encountered an issue lately. I have a cephfs cluster on 14.2.11 with 5 active MDS and 5 stand-replay MDS. Metadata pool is on SSD and datapool is on SATA. 2 of MDS restart frequently and the replay MDS stuck into replay and resolve states and never active. What's wrong with my MDS? *the restart MDS logs:* -214> 2021-03-15 17:39:00.898 7fafe3456700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -213> 2021-03-15 17:39:00.898 7fafe3456700 0 mds.beacon.aiceph-08 Skipping beacon heartbeat to monitors (last acked 54.511s ago); MDS internal heartbeat is not healthy! -212> 2021-03-15 17:39:00.911 7fafdf44e700 5 mds.3.migrator export_finish [dir 0x2000e252701.010100000110* /some/dirs/ [2,head] auth{2=2} v=143307 cv=0/0 dir_auth=2,3 state= 1073742850|frozentree|exporting f(v3086 m2021-03-12 04:21:43.383480 1303=1303+0) n(v3128 rc2021-03-12 04:21:43.384480 b119066284 1303=1303+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 frozen=1 subtree=1 importing=0 replicated=1 waiter=0 authpin=0 tempexporting=1 0x55df67f36f00] ...... -204> 2021-03-15 17:39:01.398 7fafe3456700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -203> 2021-03-15 17:39:01.398 7fafe3456700 0 mds.beacon.aiceph-08 Skipping beacon heartbeat to monitors (last acked 55.011s ago); MDS internal heartbeat is not healthy! -202> 2021-03-15 17:39:01.401 7fafe5c5b700 1 mds.beacon.aiceph-08 MDS connection to Monitors appears to be laggy; 55.014s since last acked beacon -201> 2021-03-15 17:39:01.401 7fafe5c5b700 5 mds.3.78900 laggy, deferring export_prep(0x2000e252701.110110011001*) v1 -200> 2021-03-15 17:39:01.402 7fafe3c57700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15 -199> 2021-03-15 17:39:01.402 7fafe3c57700 1 mds.3.78900 skipping upkeep work because connection to Monitors appears laggy -198> 2021-03-15 17:39:01.402 7fafe5c5b700 1 mds.3.objecter ms_handle_reset 0x55dead0ba800 session 0x55dd8944e160 osd.24 -197> 2021-03-15 17:39:01.402 7fafe5c5b700 4 mgrc ms_handle_reset ms_handle_reset con 0x55e09c44b400 -196> 2021-03-15 17:39:01.402 7fafe5c5b700 4 mgrc reconnect Terminating session with v2:172.25.64.235:6800/35103 -195> 2021-03-15 17:39:01.402 7fafe8eea700 10 monclient: get_auth_request con 0x55e15190f000 auth_method 0 -194> 2021-03-15 17:39:01.402 7fafe5c5b700 4 mgrc reconnect Starting new session with [v2:172.25.64.235:6800/35103,v1:172.25.64.235:6801/35103] -193> 2021-03-15 17:39:01.402 7fafe86e9700 10 monclient: get_auth_request con 0x55dead0ba800 auth_method 0 -192> 2021-03-15 17:39:01.467 7fafe1452700 2 mds.3.cache Memory usage: total 60577760, rss 7823000, heap 315644, baseline 315644, 28365 / 2610243 inodes have caps, 33114 caps, 0.0126862 caps per inode -191> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mds.3.78900 handle_osd_map epoch 12458, 2 new blacklist entries -190> 2021-03-15 17:39:01.467 7fafe5c5b700 10 monclient: _renew_subs -189> 2021-03-15 17:39:01.467 7fafe5c5b700 10 monclient: _send_mon_message to mon.aiceph-03 at v2:172.25.72.42:3300/0 -188> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mgrc ms_handle_reset ms_handle_reset con 0x55dead0ba800 -187> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mgrc reconnect Terminating session with v2:172.25.64.235:6800/35103 -186> 2021-03-15 17:39:01.467 7fafe5c5b700 4 mgrc reconnect waiting to retry connect until 2021-03-15 17:39:02.403432 -185> 2021-03-15 17:39:01.467 7fafdf44e700 5 mds.3.migrator export_finish [dir 0x2000e252701.010100001110* /ai_root/NLP/Member/zhangpengshen/mode_repo/phoneme_recognizer/wav2letter_simple/4000h_npy/npy/ [2,head] auth{2=2} v=142958 cv=0/0 dir_auth=2,3 state= 1073742850|frozentree|exporting f(v3086 m2021-03-12 04:21:26.970694 1333=1333+0) n(v3128 rc2021-03-12 04:21:26.970694 b121920160 1333=1333+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 frozen=1 subtree=1 importing=0 replicated=1 waiter=0 authpin=0 tempexporting=1 0 x55df671ebe00] ...... -177> 2021-03-15 17:39:01.898 7fafe3456700 5 mds.beacon.aiceph-08 Sending beacon up:active seq 1364089 -176> 2021-03-15 17:39:01.898 7fafe3456700 10 monclient: _send_mon_message to mon.aiceph-03 at v2:172.25.72.42:3300/0 ...... -3> 2021-03-15 17:39:11.669 7fafdf44e700 5 mds.3.migrator export_finish [dir 0x2000e252701.010111101010* /some/dir/ [2,head] auth{2=2} v=143028 cv=143028/143028 dir_auth=2,3 state=1073742850|frozentree|exporting f(v3086 m2021-03-12 04:22:17.528035 1340=1340+0) n(v3128 rc2021-03-12 04:22:17.529035 b125945332 1340=1340+0) hs=0+0,ss=0+0 | ptrwaiter=0 request=0 frozen=1 subtree=1 importing=0 replicated=1 waiter=0 authpin=0 tempexporting=1 0x55e029a94500] -2> 2021-03-15 17:39:11.728 7fafe5c5b700 1 mds.aiceph-08 Updating MDS map to version 199401 from mon.2 -1> 2021-03-15 17:39:11.728 7fafe5c5b700 1 mds.aiceph-08 Map removed me [mds.aiceph-08{3:977486} state up:active seq 1364038 export targets 0,1,2 addr [v2:172.25.72.47:6960/1073018517,v1:172.25.72.47:6961/1073018517]] from cluster; respawning! See cluster/monitor logs for details. 0> 2021-03-15 17:39:11.728 7fafe5c5b700 1 mds.aiceph-08 respawn! *the resolve MDS logs:* 2021-03-15 17:39:12.427 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199401 from mon.0 2021-03-15 17:39:41.092 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199402 from mon.0 2021-03-15 17:39:41.111 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199403 from mon.0 2021-03-15 17:39:41.111 7f6c0b139700 1 mds.2.199403 handle_mds_map i am now mds.2.199403 2021-03-15 17:39:41.111 7f6c0b139700 1 mds.2.199403 handle_mds_map state change up:standby-replay --> up:replay 2021-03-15 17:39:41.715 7f6c0492c700 1 mds.2.199403 standby_replay_restart (final takeover pass) 2021-03-15 17:39:41.715 7f6c0492c700 1 mds.2.199403 opening purge_queue (async) 2021-03-15 17:39:41.715 7f6c0492c700 1 mds.2.199403 opening open_file_table (async) 2021-03-15 17:39:41.716 7f6c0492c700 1 mds.2.199403 Finished replaying journal 2021-03-15 17:39:41.716 7f6c0492c700 1 mds.2.199403 making mds journal writeable 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199404 from mon.0 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 handle_mds_map i am now mds.2.199403 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 handle_mds_map state change up:replay --> up:resolve 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 resolve_start 2021-03-15 17:39:42.116 7f6c0b139700 1 mds.2.199403 reopen_log 2021-03-15 17:40:53.758 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199407 from mon.0 2021-03-15 17:41:09.478 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199408 from mon.0 2021-03-15 17:42:05.671 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199409 from mon.0 2021-03-15 17:42:15.704 7f6c0b139700 1 mds.aiceph-09 Updating MDS map to version 199410 from mon.0 *the replay MDS logs:* 2021-03-15 17:37:55.079 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199400 from mon.0 2021-03-15 17:38:52.763 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199401 from mon.0 2021-03-15 17:38:52.763 7f7a6df1c700 1 mds.3.199401 handle_mds_map i am now mds.3.199401 2021-03-15 17:38:52.763 7f7a6df1c700 1 mds.3.199401 handle_mds_map state change up:standby-replay --> up:replay 2021-03-15 17:39:21.428 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199402 from mon.0 2021-03-15 17:39:21.447 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199403 from mon.0 2021-03-15 17:39:22.452 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199404 from mon.0 2021-03-15 17:40:34.093 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199407 from mon.0 2021-03-15 17:40:49.813 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199408 from mon.0 2021-03-15 17:41:46.006 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199409 from mon.0 2021-03-15 17:41:56.039 7f7a6df1c700 1 mds.aiceph-05 Updating MDS map to version 199410 from mon.0

1 0