- ceph-users - lists.ceph.io

Re: radosgw health check url
by Marc Roos 17 Aug '20

17 Aug '20

Yes thanks, that gives at least a 200 OK without xml. However I am still getting these. 7f011d796700 1 ====== req done req=0x562a2a7f85f0 op status=0 http_status=200 latency=0s ====== 7f7c2342a700 1 ====== starting new request req=0x55c6307765f0 ===== 7f7c2342a700 1 ====== req done req=0x55c6307765f0 op status=0 http_status=200 latency=0s ====== 7f0192499700 0 WARNING: RGWRados::log_usage(): user name empty (bucket=), skipping 7f7cc117f700 0 WARNING: RGWRados::log_usage(): user name empty (bucket=), skipping 7f0119f8f700 1 ====== starting new request req=0x562a2a8105f0 ===== -----Original Message----- To: Janne Johansson Cc: Marc Roos; ceph-users Subject: Re: [ceph-users] Re: radosgw health check url Have you tried "/swift/healthcheck"? Thanks! ––––––––– Sébastien Han Senior Principal Software Engineer, Storage Architect "Always give 100%. Unless you're giving blood." On Mon, Aug 17, 2020 at 8:58 AM Janne Johansson <icepic.dz(a)gmail.com> wrote: > > Den sön 16 aug. 2020 kl 21:18 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > > > I have just put radosgw behind haproxy, and wondered if there is a > > health check endpoint. If not maybe it should be > > > Yes, very good suggestion, both for having something where the rgw > could actually state if it "feels mostly ok" in some decent fashion, > and then of course so rgw can log something nice like "served health > check data, 200 - ok" instead of "stupid broken connection attempt failed". > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Re: radosgw health check url
by Janne Johansson 17 Aug '20

17 Aug '20

Den sön 16 aug. 2020 kl 21:18 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > I have just put radosgw behind haproxy, and wondered if there is a > health check endpoint. If not maybe it should be Yes, very good suggestion, both for having something where the rgw could actually state if it "feels mostly ok" in some decent fashion, and then of course so rgw can log something nice like "served health check data, 200 - ok" instead of "stupid broken connection attempt failed". -- May the most significant bit of your life be positive.

1 0

Large omap
by Szabo, Istvan (Agoda) 17 Aug '20

17 Aug '20

Hi, I’m suffering with our large omap object in the cluster since my colleague updated from jewel to luminous 12.2.8. This warn is still here more than a year ago. LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'default.rgw.log' Search the cluster log for 'Large omap object found' for more details. Based on the articles that I’ve read it is due to the usage logs. On the weekend I’ve set the rgw_enable_usage_log = false and restarted all the 3 rgw services. I trim the logs as some people on the internet suggested, but it is super slow. I trim the logs month by month like this: radosgw-admin usage trim --end-date=2019-09-01 --yes-i-really-mean-it This command took yesterday more than a day to finish :/ I hope at least it is not increasing anymore since I turned it to false. The luminous default thresholds are: Size threshold: 1gb Key threshold: 2000000 Mine is: 7gb and 41246592 2020-08-16 09:21:11.549128 osd.45 osd.45 10.118.191.247:6824/2262114 2 : cluster [WRN] Large omap object found. Object: 4:b172cd59:usage::usage.26:head Key count: 41246592 Size (bytes): 7736069974 Based on this command here is my large omap: for i in `ceph pg ls-by-pool default.rgw.log | tail -n +2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i query |grep num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ": 1" 4.d: 1 I’ve initiated a deep scrub after it and it is still running after 12 hours and make stuck pgs: health: HEALTH_ERR 3 stuck requests are blocked > 4096 sec. Implicated osds 45 It stuck for couple of secs, and after go back to normal, but this is continously. At the moment no impact yet, but I think it will have soon. What should I do? What is the best and easiest way to fix my issue? We have serious impact with the moving of this pg when an osd crash because I guess it is big and not healthy. Here are the omap list: for i in `rados -p default.rgw.log ls`; do echo -n “$i:“; rados -p default.rgw.log listomapkeys $i |wc -l; done > /tmp/omapkeys cat /tmp/omapkeys “obj_delete_at_hint.0000000078:“0 “meta.history:“0 “obj_delete_at_hint.0000000070:“0 “obj_delete_at_hint.0000000104:“0 “obj_delete_at_hint.0000000026:“0 “obj_delete_at_hint.0000000028:“0 “obj_delete_at_hint.0000000040:“0 “obj_delete_at_hint.0000000015:“0 “obj_delete_at_hint.0000000069:“0 “obj_delete_at_hint.0000000095:“0 “obj_delete_at_hint.0000000003:“0 “obj_delete_at_hint.0000000047:“0 “obj_delete_at_hint.0000000035:“0 “obj_delete_at_hint.0000000037:“0 “obj_delete_at_hint.0000000024:“0 “obj_delete_at_hint.0000000000:“0 “obj_delete_at_hint.0000000031:“0 “obj_delete_at_hint.0000000076:“0 “obj_delete_at_hint.0000000113:“0 “obj_delete_at_hint.0000000005:“0 “obj_delete_at_hint.0000000011:“0 “obj_delete_at_hint.0000000058:“0 “obj_delete_at_hint.0000000052:“0 “obj_delete_at_hint.0000000088:“0 “obj_delete_at_hint.0000000080:“0 “obj_delete_at_hint.0000000090:“0 “obj_delete_at_hint.0000000110:“0 “obj_delete_at_hint.0000000096:“0 “obj_delete_at_hint.0000000087:“0 “obj_delete_at_hint.0000000008:“0 “obj_delete_at_hint.0000000006:“0 “obj_delete_at_hint.0000000029:“0 “obj_delete_at_hint.0000000089:“0 “obj_delete_at_hint.0000000022:“0 “bilog.trim:“0 “obj_delete_at_hint.0000000016:“0 “obj_delete_at_hint.0000000041:“0 “obj_delete_at_hint.0000000018:“0 “obj_delete_at_hint.0000000092:“0 “obj_delete_at_hint.0000000014:“0 “obj_delete_at_hint.0000000112:“0 “obj_delete_at_hint.0000000007:“0 “obj_delete_at_hint.0000000021:“0 “obj_delete_at_hint.0000000064:“0 “obj_delete_at_hint.0000000071:“0 “obj_delete_at_hint.0000000074:“0 “obj_delete_at_hint.0000000081:“0 “obj_delete_at_hint.0000000009:“0 “obj_delete_at_hint.0000000121:“0 “obj_delete_at_hint.0000000125:“0 “obj_delete_at_hint.0000000082:“0 “obj_delete_at_hint.0000000105:“0 “obj_delete_at_hint.0000000059:“0 “obj_delete_at_hint.0000000077:“0 “obj_delete_at_hint.0000000032:“0 “obj_delete_at_hint.0000000053:“0 “obj_delete_at_hint.0000000091:“0 “obj_delete_at_hint.0000000065:“0 “obj_delete_at_hint.0000000083:“0 “obj_delete_at_hint.0000000010:“0 “obj_delete_at_hint.0000000045:“0 “obj_delete_at_hint.0000000002:“0 “obj_delete_at_hint.0000000116:“0 “obj_delete_at_hint.0000000034:“0 “obj_delete_at_hint.0000000101:“0 “obj_delete_at_hint.0000000079:“0 “obj_delete_at_hint.0000000049:“0 “obj_delete_at_hint.0000000117:“0 “obj_delete_at_hint.0000000044:“0 “obj_delete_at_hint.0000000066:“0 “obj_delete_at_hint.0000000068:“0 “obj_delete_at_hint.0000000085:“0 “obj_delete_at_hint.0000000073:“0 “obj_delete_at_hint.0000000038:“0 “obj_delete_at_hint.0000000118:“0 “obj_delete_at_hint.0000000036:“0 “obj_delete_at_hint.0000000103:“0 “obj_delete_at_hint.0000000119:“0 “obj_delete_at_hint.0000000098:“0 “obj_delete_at_hint.0000000027:“0 “obj_delete_at_hint.0000000019:“0 “obj_delete_at_hint.0000000039:“0 “obj_delete_at_hint.0000000100:“0 “obj_delete_at_hint.0000000093:“0 “obj_delete_at_hint.0000000004:“0 “obj_delete_at_hint.0000000063:“0 “obj_delete_at_hint.0000000122:“0 “obj_delete_at_hint.0000000057:“0 “obj_delete_at_hint.0000000054:“0 “obj_delete_at_hint.0000000114:“0 “data_log.0:“0 “obj_delete_at_hint.0000000012:“0 “obj_delete_at_hint.0000000084:“0 “obj_delete_at_hint.0000000043:“0 “obj_delete_at_hint.0000000111:“0 “obj_delete_at_hint.0000000048:“0 “obj_delete_at_hint.0000000020:“0 “obj_delete_at_hint.0000000099:“0 “obj_delete_at_hint.0000000056:“0 “obj_delete_at_hint.0000000072:“0 “obj_delete_at_hint.0000000062:“0 “obj_delete_at_hint.0000000115:“0 “obj_delete_at_hint.0000000033:“0 “obj_delete_at_hint.0000000025:“0 “obj_delete_at_hint.0000000123:“0 “obj_delete_at_hint.0000000108:“0 “obj_delete_at_hint.0000000094:“0 “obj_delete_at_hint.0000000060:“0 “obj_delete_at_hint.0000000109:“0 “obj_delete_at_hint.0000000013:“0 “obj_delete_at_hint.0000000051:“0 “obj_delete_at_hint.0000000106:“0 “obj_delete_at_hint.0000000017:“0 “obj_delete_at_hint.0000000023:“0 “obj_delete_at_hint.0000000107:“0 “obj_delete_at_hint.0000000046:“0 “obj_delete_at_hint.0000000067:“0 “obj_delete_at_hint.0000000086:“0 “obj_delete_at_hint.0000000050:“0 “obj_delete_at_hint.0000000102:“0 “obj_delete_at_hint.0000000124:“0 “obj_delete_at_hint.0000000001:“0 “obj_delete_at_hint.0000000126:“0 “obj_delete_at_hint.0000000055:“0 “obj_delete_at_hint.0000000061:“0 “obj_delete_at_hint.0000000097:“0 “obj_delete_at_hint.0000000042:“0 “obj_delete_at_hint.0000000075:“0 “obj_delete_at_hint.0000000120:“0 “obj_delete_at_hint.0000000030:“0 Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

radosgw (ceph ) time logging
by Marc Roos 16 Aug '20

16 Aug '20

I do not need ceph/radosgw to add the time to logs going to syslog. How to turn this off, or where to change the logging format? Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.657 7f41754e0700 1 ====== starting new request req=0x55cdb3dfe5f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.659 7f41754e0700 1 ====== req done req=0x55cdb3dfe5f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.727 7f417bced700 1 ====== starting new request req=0x55cdb2b485f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.729 7f417bced700 1 ====== req done req=0x55cdb2b485f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.987 7f83d20c3700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.989 7f83d20c3700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.620 7f83cd0b9700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.622 7f83cd0b9700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.663 7f4168cc7700 1 ====== starting new request req=0x55cdb19e45f0 ===== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 0 ERROR: client_io->complete_request() returned Connection reset by peer Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 1 ====== req done req=0x55cdb19e45f0 op status=0 http_status=200 latency=0.00199996s ======

1 0

radosgw health check url
by Marc Roos 16 Aug '20

16 Aug '20

I have just put radosgw behind haproxy, and wondered if there is a health check endpoint. If not maybe it should be, now I am getting this in the logs. Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.657 7f41754e0700 1 ====== starting new request req=0x55cdb3dfe5f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.659 7f41754e0700 1 ====== req done req=0x55cdb3dfe5f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.727 7f417bced700 1 ====== starting new request req=0x55cdb2b485f0 ===== Aug 16 21:15:24 c03 radosgw: 2020-08-16 21:15:24.729 7f417bced700 1 ====== req done req=0x55cdb2b485f0 op status=0 http_status=200 latency=0.00199996s ====== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.987 7f83d20c3700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:25 c04 radosgw: 2020-08-16 21:15:25.989 7f83d20c3700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.620 7f83cd0b9700 1 ====== starting new request req=0x561ed8a625f0 ===== Aug 16 21:15:33 c04 radosgw: 2020-08-16 21:15:33.622 7f83cd0b9700 1 ====== req done req=0x561ed8a625f0 op status=0 http_status=200 latency=0.00199992s ====== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.663 7f4168cc7700 1 ====== starting new request req=0x55cdb19e45f0 ===== Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 0 ERROR: client_io->complete_request() returned Connection reset by peer Aug 16 21:15:44 c03 radosgw: 2020-08-16 21:15:44.665 7f4168cc7700 1 ====== req done req=0x55cdb19e45f0 op status=0 http_status=200 latency=0.00199996s ======

1 0

Can't add OSD id in manual deploy
by Joshua Schaeffer 15 Aug '20

15 Aug '20

Hey all, I'm trying to deploy Ceph 15.2.4 on Ubuntu 20.04 and I am going through the manual deploy process [1]. I was able to successfully bootstrap the monitor and manager and am now trying to add the OSD's, but the `ceph-volume` command is hanging when running `ceph osd new`. It appears the command hangs for about 30 minutes and then times out with this error: user@node1:~$ sudo ceph-volume lvm create --data /dev/sdc Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9a305232-6b0a-4fde-9685-68fe6150da0e stderr: [errno 110] RADOS timed out (error connecting to the cluster) --> RuntimeError: Unable to create a new OSD id I have also tried to manually run the `ceph osd new` command, but it results in the same error. My cluster shows as up (at least the monitor and manager): user@node1:~$ sudo ceph -s cluster: id: 6aacdfe1-42e4-4106-9503-3e16e9812195 health: HEALTH_WARN 1 monitors have not enabled msgr2 Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum bllcloudceph01 (age 23h) mgr: bllcloudceph01(active, since 23h) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown And ceph appears to be listening: user@node1:~$ sudo ss -tupanl | grep ceph tcp LISTEN 0 512 10.2.22.1:6789 0.0.0.0:* users:(("ceph-mon",pid=136244,fd=25)) tcp LISTEN 0 512 10.2.22.1:6800 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=25)) tcp LISTEN 0 512 10.2.22.1:6801 0.0.0.0:* users:(("ceph-mgr",pid=138125,fd=26)) It seems like I'm missing something simple, but I can't spot it. Can anybody point me in the right direction? [1] https://docs.ceph.com/docs/master/install/manual-deployment/ -- Thanks, Joshua Schaeffer

2 6

Error adding host in ceph-iscsi
by Gesiel Galvão Bernardes 15 Aug '20

15 Aug '20

Hi, I tryng add a VMware host in Ceph ISCSI. I followed exactly as is in guide. But when add the iscsi gateway IP in "Dynamic Discovery", and "rescan adapter", not loaded the "Paths". In vmkernel.log, I receive this messages: 2020-08-15T13:50:36.166Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba64:CH:0 T:0 CN:0: Connection rx notifying failure: Fai led to Receive. State=Bound 2020-08-15T13:50:36.166Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.166Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.109:59665 R: 192.168.201.1:3260 ] 2020-08-15T13:50:36.167Z cpu21:2103927)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba64:CH:1 T:0 CN:0: Failed to receive data: Connec tion closed by peer 2020-08-15T13:50:36.167Z cpu21:2103927)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:525: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.167Z cpu21:2103927)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:526: Conn [CID: 0 L: 192.168.201.109:22890 R: 192.168.201. 2:3260] 2020-08-15T13:50:36.167Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba64:CH:1 T:0 CN:0: Connection rx notifying failure: Fai led to Receive. State=Bound 2020-08-15T13:50:36.167Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1236: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.167Z cpu21:2103927)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1237: Conn [CID: 0 L: 192.168.201.109:22890 R: 192.168.201.2:3260 ] 2020-08-15T13:50:36.419Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-08-15T13:50:36.419Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.419Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.109:59665 R: 192.168.201.1:3 260] 2020-08-15T13:50:36.424Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:699: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:2) 2020-08-15T13:50:36.424Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:700: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0] 2020-08-15T13:50:36.424Z cpu32:2104379)WARNING: iscsi_vmk: iscsivmk_StopConnection:701: Conn [CID: 0 L: 192.168.201.109:22890 R: 192.168.201.2:3 260] Can anyone help-me? Regards, Gesiel

1 0

How big mon osd down out interval could be?
by huxiaoyu＠horebdata.cn 15 Aug '20

15 Aug '20

Dear Ceph folks, Suppose i have extremely reliable OSDs, which almost never fails (of course an imaginary ideal case). The OSDs may still go UPs or DOWNs due to network issues, but i would like to make OSD never OUT and thus no need for backfilling for OSD failure. To achieve the above goal, i can change mon osd down out interval from 300s to a much large value. My question is, what is the side effect of doing so, and what is the upper limit for mon osd down out interval ? best regards, samuel huxiaoyu(a)horebdata.cn

1 0

Re: how to handle incomplete PGs
by Eugen Block 15 Aug '20

15 Aug '20

Please provide more details about your environment, otherwise it's just guessing what could have happened. Zitat von huxiaoyu(a)horebdata.cn: > I am using replicated pool, and min_size=1. I do not have any disk > failure, so i do not expect incomplete PGs, but it appeared after > OSD flaped. > > > > huxiaoyu(a)horebdata.cn > > From: Eugen Block > Date: 2020-08-15 09:39 > To: huxiaoyu > CC: ceph-users > Subject: Re: [ceph-users] how to handle incomplete PGs > Hi, > > did you wait for the backfill to complete before removing the old > drives? What is your environment? Are the affected PGs from an EC > pool? Does [1] apply to you? > > [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035743.html > > > Zitat von huxiaoyu(a)horebdata.cn: > >> Dear Ceph folks, >> >> Recently i encountered incomplete PGs when replacing an OSD node >> with new handware. I noticed multiple OSD ups and downs, and >> eventually a few PGs got stucked at PG incomplete status. >> >> Questions 1: is there a reliable way to avoid the occurence of >> incomplete PGs? >> 2: is there a good tool or scriptes to handle >> incomplete PGs without lossing data >> >> best regards, >> >> samuel >> >> >> >> huxiaoyu(a)horebdata.cn >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: how to handle incomplete PGs
by Eugen Block 15 Aug '20

15 Aug '20

Hi, did you wait for the backfill to complete before removing the old drives? What is your environment? Are the affected PGs from an EC pool? Does [1] apply to you? [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035743.html Zitat von huxiaoyu(a)horebdata.cn: > Dear Ceph folks, > > Recently i encountered incomplete PGs when replacing an OSD node > with new handware. I noticed multiple OSD ups and downs, and > eventually a few PGs got stucked at PG incomplete status. > > Questions 1: is there a reliable way to avoid the occurence of > incomplete PGs? > 2: is there a good tool or scriptes to handle > incomplete PGs without lossing data > > best regards, > > samuel > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1