Hi, I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning . "ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%) "ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+ pg 2.3 is stuck undersized for 1076.456113, current state active+undersized+ pg 2.7 is stuck undersized for 1076.456342, current state active+undersized+ pg 2.8 is stuck undersized for 1076.455920, current state active+undersized+ pg 2.a is stuck undersized for 1076.486412, current state active+undersized+ pg 2.b is stuck undersized for 1076.485975, current state active+undersized+ pg 2.f is stuck undersized for 1076.486953, current state active+undersized+ pg 2.10 is stuck undersized for 1076.486763, current state active+undersized pg 2.12 is stuck undersized for 1076.486539, current state active+undersized pg 2.13 is stuck undersized for 1075.419199, current state active+undersized pg 2.17 is stuck undersized for 1076.455424, current state active+undersized pg 2.18 is stuck undersized for 1075.419639, current state active+undersized pg 2.1a is stuck undersized for 1076.455966, current state active+undersized pg 2.1b is stuck undersized for 1076.486677, current state active+undersized pg 2.1f is stuck undersized for 1076.455572, current state active+undersized how to bring it health status OK regards Amudhan
Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <amudhan83@gmail.com>:
Hi,
I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning .
"ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%)
"ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+
How does "ceph -s" look? It should have more info on what else is wrong. -- May the most significant bit of your life be positive.
In adition to ceph -s, could you provide the output of ceph osd tree and specify what your failure domain is ? /Heðin On hós, 2019-08-29 at 13:55 +0200, Janne Johansson wrote:
Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <amudhan83@gmail.com>:
Hi,
I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning .
"ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%)
"ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+
How does "ceph -s" look? It should have more info on what else is wrong.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
output from "ceph -s " cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN Degraded data redundancy: 1141587/7723191 objects degraded (14.781%), 15 pgs degraded, 16 pgs undersized services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 3 osds: 3 up, 3 in; 16 remapped pgs data: pools: 2 pools, 64 pgs objects: 2.57 M objects, 59 GiB usage: 190 GiB used, 5.3 TiB / 5.5 TiB avail pgs: 1141587/7723191 objects degraded (14.781%) 48 active+clean 15 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfilling io: recovery: 0 B/s, 10 objects/s output from "ceph osd tree" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.45819 root default -3 1.81940 host test-node1 0 hdd 1.81940 osd.0 up 1.00000 1.00000 -5 1.81940 host test-node2 1 hdd 1.81940 osd.1 up 1.00000 1.00000 -7 1.81940 host test-node3 2 hdd 1.81940 osd.2 up 1.00000 1.00000 failure domain not configured yet, setup is 3 OSD node each with a single disk, 1 node with mon&mds&mgr running. the cluster was healthy until I run a script for creating multiple folders. regards Amudhan On Thu, Aug 29, 2019 at 5:33 PM Heðin Ejdesgaard Møller <hej@synack.fo> wrote:
In adition to ceph -s, could you provide the output of ceph osd tree and specify what your failure domain is ?
/Heðin
On hós, 2019-08-29 at 13:55 +0200, Janne Johansson wrote:
Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <amudhan83@gmail.com>:
Hi,
I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning .
"ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%)
"ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+
How does "ceph -s" look? It should have more info on what else is wrong.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What's the output of ceph osd pool ls detail On hós, 2019-08-29 at 18:06 +0530, Amudhan P wrote:
output from "ceph -s "
cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN Degraded data redundancy: 1141587/7723191 objects degraded (14.781%), 15 pgs degraded, 16 pgs undersized
services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 3 osds: 3 up, 3 in; 16 remapped pgs
data: pools: 2 pools, 64 pgs objects: 2.57 M objects, 59 GiB usage: 190 GiB used, 5.3 TiB / 5.5 TiB avail pgs: 1141587/7723191 objects degraded (14.781%) 48 active+clean 15 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfilling
io: recovery: 0 B/s, 10 objects/s
output from "ceph osd tree" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.45819 root default -3 1.81940 host test-node1 0 hdd 1.81940 osd.0 up 1.00000 1.00000 -5 1.81940 host test-node2 1 hdd 1.81940 osd.1 up 1.00000 1.00000 -7 1.81940 host test-node3 2 hdd 1.81940 osd.2 up 1.00000 1.00000
failure domain not configured yet, setup is 3 OSD node each with a single disk, 1 node with mon&mds&mgr running. the cluster was healthy until I run a script for creating multiple folders.
regards Amudhan
On Thu, Aug 29, 2019 at 5:33 PM Heðin Ejdesgaard Møller < hej@synack.fo> wrote:
In adition to ceph -s, could you provide the output of ceph osd tree and specify what your failure domain is ?
/Heðin
On hós, 2019-08-29 at 13:55 +0200, Janne Johansson wrote:
Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <
amudhan83@gmail.com>:
Hi,
I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning .
"ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%)
"ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+
How does "ceph -s" look? It should have more info on what else is wrong.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
output from "ceph osd pool ls detail" pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 74 lfor 0/64 flags hashpspool stripe_width 0 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 75 lfor 0/67 flags hashpspool stripe_width 0 application cephfs On Thu, Aug 29, 2019 at 6:13 PM Heðin Ejdesgaard Møller <hej@synack.fo> wrote:
What's the output of ceph osd pool ls detail
On hós, 2019-08-29 at 18:06 +0530, Amudhan P wrote:
output from "ceph -s "
cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN Degraded data redundancy: 1141587/7723191 objects degraded (14.781%), 15 pgs degraded, 16 pgs undersized
services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 3 osds: 3 up, 3 in; 16 remapped pgs
data: pools: 2 pools, 64 pgs objects: 2.57 M objects, 59 GiB usage: 190 GiB used, 5.3 TiB / 5.5 TiB avail pgs: 1141587/7723191 objects degraded (14.781%) 48 active+clean 15 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfilling
io: recovery: 0 B/s, 10 objects/s
output from "ceph osd tree" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.45819 root default -3 1.81940 host test-node1 0 hdd 1.81940 osd.0 up 1.00000 1.00000 -5 1.81940 host test-node2 1 hdd 1.81940 osd.1 up 1.00000 1.00000 -7 1.81940 host test-node3 2 hdd 1.81940 osd.2 up 1.00000 1.00000
failure domain not configured yet, setup is 3 OSD node each with a single disk, 1 node with mon&mds&mgr running. the cluster was healthy until I run a script for creating multiple folders.
regards Amudhan
On Thu, Aug 29, 2019 at 5:33 PM Heðin Ejdesgaard Møller < hej@synack.fo> wrote:
In adition to ceph -s, could you provide the output of ceph osd tree and specify what your failure domain is ?
/Heðin
On hós, 2019-08-29 at 13:55 +0200, Janne Johansson wrote:
Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <
amudhan83@gmail.com>:
Hi,
I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning .
"ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%)
"ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+
How does "ceph -s" look? It should have more info on what else is wrong.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, ceph uses a pseudo random distribution within crush to select the target hosts. As a result, the algorithm might not be able to select three different hosts out of three hosts in the configured number of tries. The affected PGs will be shown as undersized and only list two OSDs instead of three. The best solution is adding more host, the second best is changing crush parameters (number_of_tries etc.) until the algorithm gives better results. Ceph clusters with few hosts often require manual tuning to work reliably. Regards, Burkhard
Hi, This output doesn't show anything 'wrong' with the cluster. It's just still recovering (backfilling) from what seems like one of your OSD's crashed and restarted. The backfilling is taking a while because max_backfills = 1 and you only have 3 OSD's total so the backfilling per PG has to have for the previous PG backfill to complete. The real concern is not the current state of the cluster but how you end up in this state. Probably the script overloaded the OSD's. I also advise you to add a monitor to your other 2 nodes as well (running 3 mons total). Running 1 mon is not advised. Furthermore, just let the backfilling complete and HEALTH_OK will return eventually if nothing goes wrong in between. Met vriendelijke groet, Caspar Smit Systemengineer SuperNAS Dorsvlegelstraat 13 1445 PA Purmerend t: (+31) 299 410 414 e: casparsmit@supernas.eu w: www.supernas.eu Op do 29 aug. 2019 om 14:35 schreef Amudhan P <amudhan83@gmail.com>:
output from "ceph -s "
cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN Degraded data redundancy: 1141587/7723191 objects degraded (14.781%), 15 pgs degraded, 16 pgs undersized
services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 3 osds: 3 up, 3 in; 16 remapped pgs
data: pools: 2 pools, 64 pgs objects: 2.57 M objects, 59 GiB usage: 190 GiB used, 5.3 TiB / 5.5 TiB avail pgs: 1141587/7723191 objects degraded (14.781%) 48 active+clean 15 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfilling
io: recovery: 0 B/s, 10 objects/s
output from "ceph osd tree" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.45819 root default -3 1.81940 host test-node1 0 hdd 1.81940 osd.0 up 1.00000 1.00000 -5 1.81940 host test-node2 1 hdd 1.81940 osd.1 up 1.00000 1.00000 -7 1.81940 host test-node3 2 hdd 1.81940 osd.2 up 1.00000 1.00000
failure domain not configured yet, setup is 3 OSD node each with a single disk, 1 node with mon&mds&mgr running. the cluster was healthy until I run a script for creating multiple folders.
regards Amudhan
On Thu, Aug 29, 2019 at 5:33 PM Heðin Ejdesgaard Møller <hej@synack.fo> wrote:
In adition to ceph -s, could you provide the output of ceph osd tree and specify what your failure domain is ?
/Heðin
On hós, 2019-08-29 at 13:55 +0200, Janne Johansson wrote:
Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <amudhan83@gmail.com>:
Hi,
I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning .
"ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%)
"ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+
How does "ceph -s" look? It should have more info on what else is wrong.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
After leaving 12 hours time now cluster status is healthy, but why did it take such a long time for backfill? How do I fine-tune? if in case of same kind error pop-out again. On Thu, Aug 29, 2019 at 6:52 PM Caspar Smit <casparsmit@supernas.eu> wrote:
Hi,
This output doesn't show anything 'wrong' with the cluster. It's just still recovering (backfilling) from what seems like one of your OSD's crashed and restarted. The backfilling is taking a while because max_backfills = 1 and you only have 3 OSD's total so the backfilling per PG has to have for the previous PG backfill to complete.
The real concern is not the current state of the cluster but how you end up in this state. Probably the script overloaded the OSD's.
I also advise you to add a monitor to your other 2 nodes as well (running 3 mons total). Running 1 mon is not advised.
Furthermore, just let the backfilling complete and HEALTH_OK will return eventually if nothing goes wrong in between.
Met vriendelijke groet,
Caspar Smit Systemengineer SuperNAS Dorsvlegelstraat 13 1445 PA Purmerend
t: (+31) 299 410 414 e: casparsmit@supernas.eu w: www.supernas.eu
Op do 29 aug. 2019 om 14:35 schreef Amudhan P <amudhan83@gmail.com>:
output from "ceph -s "
cluster: id: 7c138e13-7b98-4309-b591-d4091a1742b4 health: HEALTH_WARN Degraded data redundancy: 1141587/7723191 objects degraded (14.781%), 15 pgs degraded, 16 pgs undersized
services: mon: 1 daemons, quorum mon01 mgr: mon01(active) mds: cephfs-tst-1/1/1 up {0=mon01=up:active} osd: 3 osds: 3 up, 3 in; 16 remapped pgs
data: pools: 2 pools, 64 pgs objects: 2.57 M objects, 59 GiB usage: 190 GiB used, 5.3 TiB / 5.5 TiB avail pgs: 1141587/7723191 objects degraded (14.781%) 48 active+clean 15 active+undersized+degraded+remapped+backfill_wait 1 active+undersized+remapped+backfilling
io: recovery: 0 B/s, 10 objects/s
output from "ceph osd tree" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.45819 root default -3 1.81940 host test-node1 0 hdd 1.81940 osd.0 up 1.00000 1.00000 -5 1.81940 host test-node2 1 hdd 1.81940 osd.1 up 1.00000 1.00000 -7 1.81940 host test-node3 2 hdd 1.81940 osd.2 up 1.00000 1.00000
failure domain not configured yet, setup is 3 OSD node each with a single disk, 1 node with mon&mds&mgr running. the cluster was healthy until I run a script for creating multiple folders.
regards Amudhan
On Thu, Aug 29, 2019 at 5:33 PM Heðin Ejdesgaard Møller <hej@synack.fo> wrote:
In adition to ceph -s, could you provide the output of ceph osd tree and specify what your failure domain is ?
/Heðin
On hós, 2019-08-29 at 13:55 +0200, Janne Johansson wrote:
Den tors 29 aug. 2019 kl 13:50 skrev Amudhan P <amudhan83@gmail.com>:
Hi,
I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs. my ceph health status showing warning .
"ceph health" HEALTH_WARN Degraded data redundancy: 1197023/7723191 objects degraded (15.499%)
"ceph health detail" HEALTH_WARN Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) PG_DEGRADED Degraded data redundancy: 1197128/7723191 objects degraded (15.500%) pg 2.0 is stuck undersized for 1076.454929, current state active+undersized+ pg 2.2 is stuck undersized for 1076.456639, current state active+undersized+
How does "ceph -s" look? It should have more info on what else is wrong.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
After leaving 12 hours time now cluster status is healthy, but why did it take such a long time for backfill? How do I fine-tune? if in case of same kind error pop-out again.
The backfilling is taking a while because max_backfills = 1 and you only
have 3 OSD's total so the backfilling per PG has to have for the previous PG backfill to complete.
That setting is the main tuning, EXCEPT it will be at the expense of client
Den fre 30 aug. 2019 kl 10:49 skrev Amudhan P <amudhan83@gmail.com>: traffic, so you can allow a large(r) amount of parallel recoveries and backfills, but of course it will be more noticeable for your client IO if you do. Lastly, getting backfill MB/s up is "best" done by having a huge amount of OSD hosts, and fast OSD drives and let the cluster work in parallel, as opposed to having 3 drives only because you will see no parallelism on that setup (if you have size=3 all OSDs are always involved in every single PG to recover) and you will just see overhead compare to what disk-read and disk-write would give on a single drive. -- May the most significant bit of your life be positive.
my cluster health status went to warning mode only after running mkdir of 1000's of folders with multiple subdirectories. if this has made OSD crash does it really takes that long to heal empty directories. On Fri, Aug 30, 2019 at 3:12 PM Janne Johansson <icepic.dz@gmail.com> wrote:
Den fre 30 aug. 2019 kl 10:49 skrev Amudhan P <amudhan83@gmail.com>:
After leaving 12 hours time now cluster status is healthy, but why did it take such a long time for backfill? How do I fine-tune? if in case of same kind error pop-out again.
The backfilling is taking a while because max_backfills = 1 and you only
have 3 OSD's total so the backfilling per PG has to have for the previous PG backfill to complete.
That setting is the main tuning, EXCEPT it will be at the expense of client traffic, so you can allow a large(r) amount of parallel recoveries and backfills, but of course it will be more noticeable for your client IO if you do.
Lastly, getting backfill MB/s up is "best" done by having a huge amount of OSD hosts, and fast OSD drives and let the cluster work in parallel, as opposed to having 3 drives only because you will see no parallelism on that setup (if you have size=3 all OSDs are always involved in every single PG to recover) and you will just see overhead compare to what disk-read and disk-write would give on a single drive.
-- May the most significant bit of your life be positive.
participants (5)
-
Amudhan P
-
Burkhard Linke
-
Caspar Smit
-
Heðin Ejdesgaard Møller
-
Janne Johansson