Ceph MDS randomly hangs when pg nums reduced
Hi, I have a CephFS cluster ```
ceph -s
cluster: id: e78987f2-ef1c-11ed-897d-cf8c255417f0 health: HEALTH_WARN 85 pgs not deep-scrubbed in time 85 pgs not scrubbed in time services: mon: 5 daemons, quorum datastone05,datastone06,datastone07,datastone10,datastone09 (age 2w) mgr: datastone05.iitngk(active, since 2w), standbys: datastone06.wjppdy mds: 2/2 daemons up, 1 hot standby osd: 22 osds: 22 up (since 3d), 22 in (since 4w); 8 remapped pgs data: volumes: 1/1 healthy pools: 4 pools, 115 pgs objects: 49.08M objects, 16 TiB usage: 35 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 3807933/98160678 objects misplaced (3.879%) 107 active+clean 8 active+remapped+backfilling io: client: 224 MiB/s rd, 79 MiB/s wr, 844 op/s rd, 33 op/s wr recovery: 8.8 MiB/s, 24 objects/s ``` The pool and pg status ```
ceph osd pool autoscale-status
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK cephfs.myfs.meta 28802M 2.0 2119T 0.0000 4.0 16 on False cephfs.myfs.data 16743G 2.0 2119T 0.0154 1.0 32 on False rbd 19 2.0 2119T 0.0000 1.0 32 on False .mgr 3840k 2.0 2119T 0.0000 1.0 1 on False ``` The pool detail ```
ceph osd pool ls detail
pool 1 'cephfs.myfs.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 3639 lfor 0/3639/3637 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 2 'cephfs.myfs.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 66 pgp_num 58 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 5670 lfor 0/5661/5659 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 486 lfor 0/486/478 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 4 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 39 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr ``` When pg numbers reduce, the mds server would have a chance to hang.
Hi, First thing that comes to my mind looking at the I/O section is the increased load of metadata distribution causing bottlenecks to the MDSs, but you also have 2 active ranks which could avert the problem but then it could be possible that certain files/dirs get to do unreasonably high amount of metadata I/O which could've lead to uneven distribution of workload among those active MDSs therefore making one of them hang at times. If the MDS is hanging it seems like yours is a metadata intensive environment(like CephFS) and reducing PGs might not be a good idea here. You could also share MDS logs to see what's going on exactly and if there is something that needs attention. -- *Dhairya Parmar* Associate Software Engineer, CephFS IBM, Inc. On Fri, Feb 23, 2024 at 8:27 PM <lokitingyi@gmail.com> wrote:
Hi,
I have a CephFS cluster ```
ceph -s
cluster: id: e78987f2-ef1c-11ed-897d-cf8c255417f0 health: HEALTH_WARN 85 pgs not deep-scrubbed in time 85 pgs not scrubbed in time
services: mon: 5 daemons, quorum datastone05,datastone06,datastone07,datastone10,datastone09 (age 2w) mgr: datastone05.iitngk(active, since 2w), standbys: datastone06.wjppdy mds: 2/2 daemons up, 1 hot standby osd: 22 osds: 22 up (since 3d), 22 in (since 4w); 8 remapped pgs
data: volumes: 1/1 healthy pools: 4 pools, 115 pgs objects: 49.08M objects, 16 TiB usage: 35 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 3807933/98160678 objects misplaced (3.879%) 107 active+clean 8 active+remapped+backfilling
io: client: 224 MiB/s rd, 79 MiB/s wr, 844 op/s rd, 33 op/s wr recovery: 8.8 MiB/s, 24 objects/s ```
The pool and pg status
```
ceph osd pool autoscale-status
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK cephfs.myfs.meta 28802M 2.0 2119T 0.0000 4.0 16 on False cephfs.myfs.data 16743G 2.0 2119T 0.0154 1.0 32 on False rbd 19 2.0 2119T 0.0000 1.0 32 on False .mgr 3840k 2.0 2119T 0.0000 1.0 1 on False ```
The pool detail
```
ceph osd pool ls detail
pool 1 'cephfs.myfs.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 3639 lfor 0/3639/3637 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 2 'cephfs.myfs.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 66 pgp_num 58 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 5670 lfor 0/5661/5659 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 486 lfor 0/486/478 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 4 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 39 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr ```
When pg numbers reduce, the mds server would have a chance to hang. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Dhairya Parmar
-
lokitingyi＠gmail.com