Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <amudhan83(a)gmail.com>om>: How does "ceph -s" look? It should have more info on what else is wrong. -- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

output from "ceph -s " cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN Degraded data redundancy: 1141587/7723191 objects degraded (14.781%), 15 pgs degraded, 16 pgs undersized services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 3 osds: 3 up, 3 in; 16 remapped pgs data: pools: 2 pools, 64 pgs objects: 2.57 M objects, 59 GiB usage: 190 GiB used, 5.3 TiB / 5.5 TiB avail pgs: 1141587/7723191 objects degraded (14.781%) 48 active+clean 15 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfilling io: recovery: 0 B/s, 10 objects/s output from "ceph osd tree" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.45819 root default -3 1.81940 host test-node1 0 hdd 1.81940 osd.0 up 1.00000 1.00000 -5 1.81940 host test-node2 1 hdd 1.81940 osd.1 up 1.00000 1.00000 -7 1.81940 host test-node3 2 hdd 1.81940 osd.2 up 1.00000 1.00000 failure domain not configured yet, setup is 3 OSD node each with a single disk, 1 node with mon&mds&mgr running. the cluster was healthy until I run a script for creating multiple folders. regards Amudhan On Thu, Aug 29, 2019 at 5:33 PM Heðin Ejdesgaard Møller < hej(a)synack.fo&gt; wrote: > In adition to ceph -s, could you provide the output of > ceph osd tree > and specify what your failure domain is ? > > /Heðin > > > On hós, 2019-08-29 at 13:55 +0200, Janne Johansson wrote: > > > > > > Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P < > amudhan83(a)gmail.com&gt;gt;: > > > Hi, > > > > > > I am using ceph version 13.2.6 (mimic) on test setup trying > with > > > cephfs. > > > my ceph health status showing warning . > > > > > > "ceph health" > > > HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects > > > degraded (15.499%) > > > > > > "ceph health detail" > > > HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects > > > degraded (15.500%) > > > PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects > > > degraded (15.500%) > > > pg 2.0 is stuck undersized for 1076.454929, current state > > > active+undersized+ > > > pg 2.2 is stuck undersized for 1076.456639, current state > > > active+undersized+ > > > > > > > How does "ceph -s" look? > > It should have more info on what else is wrong. > > > > -- > > May the most significant bit of your life be positive. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

Hi, This output doesn't show anything 'wrong' with the cluster. It's just still recovering (backfilling) from what seems like one of your OSD's crashed and restarted. The backfilling is taking a while because max_backfills = 1 and you only have 3 OSD's total so the backfilling per PG has to have for the previous PG backfill to complete. The real concern is not the current state of the cluster but how you end up in this state. Probably the script overloaded the OSD's. I also advise you to add a monitor to your other 2 nodes as well (running 3 mons total). Running 1 mon is not advised. Furthermore, just let the backfilling complete and HEALTH_OK will return eventually if nothing goes wrong in between. Met vriendelijke groet, Caspar Smit Systemengineer SuperNAS Dorsvlegelstraat 13 1445 PA Purmerend t: (+31) 299 410 414 e: casparsmit(a)supernas.eu w: www.supernas.eu Op do 29 aug. 2019 om 14:35 schreef Amudhan P &lt;amudhan83(a)gmail.com&gt;om>: _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

Den fre 30 aug. 2019 kl 10:49 skrev Amudhan P &lt;amudhan83(a)gmail.com&gt;om>: That setting is the main tuning, EXCEPT it will be at the expense of client traffic, so you can allow a large(r) amount of parallel recoveries and backfills, but of course it will be more noticeable for your client IO if you do. Lastly, getting backfill MB/s up is "best" done by having a huge amount of OSD hosts, and fast OSD drives and let the cluster work in parallel, as opposed to having 3 drives only because you will see no parallelism on that setup (if you have size=3 all OSDs are always involved in every single PG to recover) and you will just see overhead compare to what disk-read and disk-write would give on a single drive. -- May the most significant bit of your life be positive.