Cannot start virtual machines KVM / LXC
Hi, here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS). The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox. However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked". When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour. Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd. Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd What can / should I do now? Kill the long running processes? Stop the relevant OSDs? Please advise? THX Thomas
Your need to fix this first. pgs: 0.056% pgs unknown 0.553% pgs not active The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting. Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online. ---- On Fri, 20 Sep 2019 14:14:01 +0800 Thomas <74cmonty@gmail.com> wrote ---- Hi, here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS). The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox. However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked". When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour. Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd. Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd What can / should I do now? Kill the long running processes? Stop the relevant OSDs? Please advise? THX Thomas _______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io
Hi, I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks. Then I added the 3 OSDs to crushmap. Regards Thomas Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks.
and you had it configured to run with replica 3? Well, I guess the down PGs where located on these three disks that you wiped. Do you still have the disks? Use ceph-objectstore-tool to export the affected PGs manually and inject them into another OSD. Paul
Then I added the 3 OSDs to crushmap.
Regards Thomas
Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, I have re-created the OSDs using these disks. Can I still export the affected PGs manually? Regards Thomas Am 20.09.19 um 21:15 schrieb Paul Emmerich:
On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks. and you had it configured to run with replica 3? Well, I guess the down PGs where located on these three disks that you wiped.
Do you still have the disks? Use ceph-objectstore-tool to export the affected PGs manually and inject them into another OSD.
Paul
Then I added the 3 OSDs to crushmap.
Regards Thomas
Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Sat, Sep 21, 2019 at 6:47 PM Thomas <74cmonty@gmail.com> wrote:
Hello,
I have re-created the OSDs using these disks. Can I still export the affected PGs manually?
No, the data probably cannot be recovered in this case :( (It might still be somewhere on the disk if it hasn't been overwritten yet, but it's virtually impossible to recover it: the metadata has almost certainly long been overwritten) But that's only for 17 PGs showing up as unknown. The ones stuck in peering can probably be revived but without the latest writes. Can you run "ceph pg X.YX query" on one of the PGs stuck in peering? It might tell you what's wrong and how to proceed. But even 17 PGs will probably affect almost all of your VMs/containers... Paul
Regards Thomas
Am 20.09.19 um 21:15 schrieb Paul Emmerich:
On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks. and you had it configured to run with replica 3? Well, I guess the down PGs where located on these three disks that you wiped.
Do you still have the disks? Use ceph-objectstore-tool to export the affected PGs manually and inject them into another OSD.
Paul
Then I added the 3 OSDs to crushmap.
Regards Thomas
Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, currently ceph -s is not reporting any unknown PGs. The following flags are set: nobackfill, norebalance, norecover There are no PGs in stuck peering either. And there's very little traffic on the ceph network. This is the output of today: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available nobackfill,norebalance,norecover flag(s) set 83 nearfull osd(s) 1 pool(s) nearfull Degraded data redundancy: 360047/153249771 objects degraded (0.235%), 78 pgs degraded, 81 pgs undersized Degraded data redundancy (low space): 265 pgs backfill_toofull 3 pools have too many placement groups services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 109m) mgr: ld5505(active, since 2d), standbys: ld5506, ld5507 mds: pve_cephfs:0/1, 1 failed osd: 368 osds: 368 up, 367 in; 398 remapped pgs flags nobackfill,norebalance,norecover data: pools: 5 pools, 8868 pgs objects: 51.08M objects, 195 TiB usage: 590 TiB used, 562 TiB / 1.1 PiB avail pgs: 360047/153249771 objects degraded (0.235%) 1998603/153249771 objects misplaced (1.304%) 8469 active+clean 124 active+remapped+backfill_toofull 83 active+remapped+backfill_wait+backfill_toofull 77 active+remapped+backfill_wait 45 active+undersized+degraded+remapped+backfill_toofull 33 active+remapped+backfilling 13 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 11 active+undersized+degraded+remapped+backfilling 4 active+undersized+degraded+remapped+backfill_wait 4 active+recovery_wait+undersized+degraded+remapped 3 active+recovering+undersized+remapped 1 active+recovering+undersized+degraded+remapped 1 active+recovering io: client: 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr progress: Rebalancing after osd.9 marked out [====================..........] 2 days before the output was: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available nobackfill,norebalance,norecover flag(s) set 2 backfillfull osd(s) 86 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 75 pgs inactive, 74 pgs peering Degraded data redundancy: 364117/154942251 objects degraded (0.235%), 76 pgs degraded, 76 pgs undersized Degraded data redundancy (low space): 309 pgs backfill_toofull 3 pools have too many placement groups 105 slow requests are blocked > 32 sec 91 stuck requests are blocked > 4096 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 9h) mgr: ld5505(active, since 9h), standbys: ld5506, ld5507 mds: pve_cephfs:0/1, 1 failed osd: 368 osds: 368 up, 367 in; 400 remapped pgs flags nobackfill,norebalance,norecover data: pools: 5 pools, 8868 pgs objects: 51.65M objects, 197 TiB usage: 596 TiB used, 554 TiB / 1.1 PiB avail pgs: 0.011% pgs unknown 0.834% pgs not active 364117/154942251 objects degraded (0.235%) 2003579/154942251 objects misplaced (1.293%) 8395 active+clean 209 active+remapped+backfill_toofull 69 active+undersized+degraded+remapped+backfill_toofull 49 active+remapped+backfill_wait 37 peering 37 remapped+peering 31 active+remapped+backfill_wait+backfill_toofull 17 active+clean+scrubbing+deep 14 active+clean+scrubbing 4 active+undersized+degraded 2 active+undersized+degraded+remapped+backfill_wait 1 unknown 1 active+clean+remapped 1 active+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling io: client: 256 MiB/s wr, 0 op/s rd, 99 op/s wr progress: Rebalancing after osd.9 marked out [==............................] As you can see there's very little progress in "Rebalancing after osd.9 marked out", the number of objects degraded dropped to 360047/153249771 and the number of objects misplaced to 1998603/153249771. I think this is very little progress in 2 days with no activity (over the weekend) on the cluster. Am 21.09.2019 um 20:39 schrieb Paul Emmerich:
On Sat, Sep 21, 2019 at 6:47 PM Thomas <74cmonty@gmail.com> wrote:
Hello,
I have re-created the OSDs using these disks. Can I still export the affected PGs manually? No, the data probably cannot be recovered in this case :( (It might still be somewhere on the disk if it hasn't been overwritten yet, but it's virtually impossible to recover it: the metadata has almost certainly long been overwritten)
But that's only for 17 PGs showing up as unknown. The ones stuck in peering can probably be revived but without the latest writes. Can you run "ceph pg X.YX query" on one of the PGs stuck in peering? It might tell you what's wrong and how to proceed.
But even 17 PGs will probably affect almost all of your VMs/containers...
Paul
Regards Thomas
Am 20.09.19 um 21:15 schrieb Paul Emmerich:
On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks. and you had it configured to run with replica 3? Well, I guess the down PGs where located on these three disks that you wiped.
Do you still have the disks? Use ceph-objectstore-tool to export the affected PGs manually and inject them into another OSD.
Paul
Then I added the 3 OSDs to crushmap.
Regards Thomas
Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Have you been able to start the VMs now? Are you using RBD or are the VM's hosted on a CephFS? ---- On Mon, 23 Sep 2019 14:16:47 +0800 Thomas Schneider <74cmonty@gmail.com> wrote ---- Hi, currently ceph -s is not reporting any unknown PGs. The following flags are set: nobackfill, norebalance, norecover There are no PGs in stuck peering either. And there's very little traffic on the ceph network. This is the output of today: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available nobackfill,norebalance,norecover flag(s) set 83 nearfull osd(s) 1 pool(s) nearfull Degraded data redundancy: 360047/153249771 objects degraded (0.235%), 78 pgs degraded, 81 pgs undersized Degraded data redundancy (low space): 265 pgs backfill_toofull 3 pools have too many placement groups services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 109m) mgr: ld5505(active, since 2d), standbys: ld5506, ld5507 mds: pve_cephfs:0/1, 1 failed osd: 368 osds: 368 up, 367 in; 398 remapped pgs flags nobackfill,norebalance,norecover data: pools: 5 pools, 8868 pgs objects: 51.08M objects, 195 TiB usage: 590 TiB used, 562 TiB / 1.1 PiB avail pgs: 360047/153249771 objects degraded (0.235%) 1998603/153249771 objects misplaced (1.304%) 8469 active+clean 124 active+remapped+backfill_toofull 83 active+remapped+backfill_wait+backfill_toofull 77 active+remapped+backfill_wait 45 active+undersized+degraded+remapped+backfill_toofull 33 active+remapped+backfilling 13 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 11 active+undersized+degraded+remapped+backfilling 4 active+undersized+degraded+remapped+backfill_wait 4 active+recovery_wait+undersized+degraded+remapped 3 active+recovering+undersized+remapped 1 active+recovering+undersized+degraded+remapped 1 active+recovering io: client: 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr progress: Rebalancing after osd.9 marked out [====================..........] 2 days before the output was: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available nobackfill,norebalance,norecover flag(s) set 2 backfillfull osd(s) 86 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 75 pgs inactive, 74 pgs peering Degraded data redundancy: 364117/154942251 objects degraded (0.235%), 76 pgs degraded, 76 pgs undersized Degraded data redundancy (low space): 309 pgs backfill_toofull 3 pools have too many placement groups 105 slow requests are blocked > 32 sec 91 stuck requests are blocked > 4096 sec services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 9h) mgr: ld5505(active, since 9h), standbys: ld5506, ld5507 mds: pve_cephfs:0/1, 1 failed osd: 368 osds: 368 up, 367 in; 400 remapped pgs flags nobackfill,norebalance,norecover data: pools: 5 pools, 8868 pgs objects: 51.65M objects, 197 TiB usage: 596 TiB used, 554 TiB / 1.1 PiB avail pgs: 0.011% pgs unknown 0.834% pgs not active 364117/154942251 objects degraded (0.235%) 2003579/154942251 objects misplaced (1.293%) 8395 active+clean 209 active+remapped+backfill_toofull 69 active+undersized+degraded+remapped+backfill_toofull 49 active+remapped+backfill_wait 37 peering 37 remapped+peering 31 active+remapped+backfill_wait+backfill_toofull 17 active+clean+scrubbing+deep 14 active+clean+scrubbing 4 active+undersized+degraded 2 active+undersized+degraded+remapped+backfill_wait 1 unknown 1 active+clean+remapped 1 active+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling io: client: 256 MiB/s wr, 0 op/s rd, 99 op/s wr progress: Rebalancing after osd.9 marked out [==............................] As you can see there's very little progress in "Rebalancing after osd.9 marked out", the number of objects degraded dropped to 360047/153249771 and the number of objects misplaced to 1998603/153249771. I think this is very little progress in 2 days with no activity (over the weekend) on the cluster. Am 21.09.2019 um 20:39 schrieb Paul Emmerich:
On Sat, Sep 21, 2019 at 6:47 PM Thomas <mailto:74cmonty@gmail.com> wrote:
Hello,
I have re-created the OSDs using these disks. Can I still export the affected PGs manually? No, the data probably cannot be recovered in this case :( (It might still be somewhere on the disk if it hasn't been overwritten yet, but it's virtually impossible to recover it: the metadata has almost certainly long been overwritten)
But that's only for 17 PGs showing up as unknown. The ones stuck in peering can probably be revived but without the latest writes. Can you run "ceph pg X.YX query" on one of the PGs stuck in peering? It might tell you what's wrong and how to proceed.
But even 17 PGs will probably affect almost all of your VMs/containers...
Paul
Regards Thomas
Am 20.09.19 um 21:15 schrieb Paul Emmerich:
On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider <mailto:74cmonty@gmail.com> wrote:
Hi,
I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks. and you had it configured to run with replica 3? Well, I guess the down PGs where located on these three disks that you wiped.
Do you still have the disks? Use ceph-objectstore-tool to export the affected PGs manually and inject them into another OSD.
Paul
Then I added the 3 OSDs to crushmap.
Regards Thomas
Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <mailto:74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io <mailto:mailto:ceph-users@ceph.io> To unsubscribe send an email to mailto:ceph-users-leave@ceph.io <mailto:mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- mailto:ceph-users@ceph.io To unsubscribe send an email to mailto:ceph-users-leave@ceph.io
None of the KVM / LXC instances is starting. Any KVM / LXC instance is using RBD. The same pool hdd is providing CephFS service, but this is only used for storing KVM / LXC instance backups, ISOs and other files. Am 23.09.2019 um 08:55 schrieb Ashley Merrick:
Have you been able to start the VMs now?
Are you using RBD or are the VM's hosted on a CephFS?
---- On Mon, 23 Sep 2019 14:16:47 +0800 *Thomas Schneider <74cmonty@gmail.com>* wrote ----
Hi,
currently ceph -s is not reporting any unknown PGs. The following flags are set: nobackfill, norebalance, norecover There are no PGs in stuck peering either. And there's very little traffic on the ceph network.
This is the output of today: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available nobackfill,norebalance,norecover flag(s) set 83 nearfull osd(s) 1 pool(s) nearfull Degraded data redundancy: 360047/153249771 objects degraded (0.235%), 78 pgs degraded, 81 pgs undersized Degraded data redundancy (low space): 265 pgs backfill_toofull 3 pools have too many placement groups
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 109m) mgr: ld5505(active, since 2d), standbys: ld5506, ld5507 mds: pve_cephfs:0/1, 1 failed osd: 368 osds: 368 up, 367 in; 398 remapped pgs flags nobackfill,norebalance,norecover
data: pools: 5 pools, 8868 pgs objects: 51.08M objects, 195 TiB usage: 590 TiB used, 562 TiB / 1.1 PiB avail pgs: 360047/153249771 objects degraded (0.235%) 1998603/153249771 objects misplaced (1.304%) 8469 active+clean 124 active+remapped+backfill_toofull 83 active+remapped+backfill_wait+backfill_toofull 77 active+remapped+backfill_wait 45 active+undersized+degraded+remapped+backfill_toofull 33 active+remapped+backfilling 13 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 11 active+undersized+degraded+remapped+backfilling 4 active+undersized+degraded+remapped+backfill_wait 4 active+recovery_wait+undersized+degraded+remapped 3 active+recovering+undersized+remapped 1 active+recovering+undersized+degraded+remapped 1 active+recovering
io: client: 5.6 KiB/s wr, 0 op/s rd, 0 op/s wr
progress: Rebalancing after osd.9 marked out [====================..........]
2 days before the output was: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 filesystem has a failed mds daemon 1 filesystem is offline insufficient standby MDS daemons available nobackfill,norebalance,norecover flag(s) set 2 backfillfull osd(s) 86 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 75 pgs inactive, 74 pgs peering Degraded data redundancy: 364117/154942251 objects degraded (0.235%), 76 pgs degraded, 76 pgs undersized Degraded data redundancy (low space): 309 pgs backfill_toofull 3 pools have too many placement groups 105 slow requests are blocked > 32 sec 91 stuck requests are blocked > 4096 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 9h) mgr: ld5505(active, since 9h), standbys: ld5506, ld5507 mds: pve_cephfs:0/1, 1 failed osd: 368 osds: 368 up, 367 in; 400 remapped pgs flags nobackfill,norebalance,norecover
data: pools: 5 pools, 8868 pgs objects: 51.65M objects, 197 TiB usage: 596 TiB used, 554 TiB / 1.1 PiB avail pgs: 0.011% pgs unknown 0.834% pgs not active 364117/154942251 objects degraded (0.235%) 2003579/154942251 objects misplaced (1.293%) 8395 active+clean 209 active+remapped+backfill_toofull 69 active+undersized+degraded+remapped+backfill_toofull 49 active+remapped+backfill_wait 37 peering 37 remapped+peering 31 active+remapped+backfill_wait+backfill_toofull 17 active+clean+scrubbing+deep 14 active+clean+scrubbing 4 active+undersized+degraded 2 active+undersized+degraded+remapped+backfill_wait 1 unknown 1 active+clean+remapped 1 active+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling
io: client: 256 MiB/s wr, 0 op/s rd, 99 op/s wr
progress: Rebalancing after osd.9 marked out [==............................]
As you can see there's very little progress in "Rebalancing after osd.9 marked out", the number of objects degraded dropped to 360047/153249771 and the number of objects misplaced to 1998603/153249771.
I think this is very little progress in 2 days with no activity (over the weekend) on the cluster.
Am 21.09.2019 um 20:39 schrieb Paul Emmerich: > On Sat, Sep 21, 2019 at 6:47 PM Thomas <74cmonty@gmail.com <mailto:74cmonty@gmail.com>> wrote: >> Hello, >> >> I have re-created the OSDs using these disks. >> Can I still export the affected PGs manually? > No, the data probably cannot be recovered in this case :( > (It might still be somewhere on the disk if it hasn't been overwritten > yet, but it's virtually impossible to recover it: the metadata has > almost certainly long been overwritten) > > But that's only for 17 PGs showing up as unknown. The ones stuck in > peering can probably be revived but without the latest writes. Can you > run "ceph pg X.YX query" on one of the PGs stuck in peering? > It might tell you what's wrong and how to proceed. > > But even 17 PGs will probably affect almost all of your VMs/containers... > > Paul > >> Regards >> Thomas >> >> >> Am 20.09.19 um 21:15 schrieb Paul Emmerich: >>> On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider <74cmonty@gmail.com <mailto:74cmonty@gmail.com>> wrote: >>>> Hi, >>>> >>>> I cannot get rid of >>>> pgs unknown >>>> because there were 3 disks that couldn't be started. >>>> Therefore I destroyed the relevant OSD and re-created it for the >>>> relevant disks. >>> and you had it configured to run with replica 3? Well, I guess the >>> down PGs where located on these three disks that you wiped. >>> >>> Do you still have the disks? Use ceph-objectstore-tool to export the >>> affected PGs manually and inject them into another OSD. >>> >>> >>> Paul >>> >>>> Then I added the 3 OSDs to crushmap. >>>> >>>> Regards >>>> Thomas >>>> >>>> Am 20.09.2019 um 08:19 schrieb Ashley Merrick: >>>>> Your need to fix this first. >>>>> >>>>> pgs: 0.056% pgs unknown >>>>> 0.553% pgs not active >>>>> >>>>> The back filling will cause slow I/O, but having pgs unknown and not >>>>> active will cause I/O blocking which your seeing with the VM booting. >>>>> >>>>> Seems you have 4 OSD's down, if you get them back online you should be >>>>> able to get all the PG's online. >>>>> >>>>> >>>>> ---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com <mailto:74cmonty@gmail.com>>* >>>>> wrote ---- >>>>> >>>>> Hi, >>>>> >>>>> here I describe 1 of the 2 major issues I'm currently facing in my 8 >>>>> node ceph cluster (2x MDS, 6x ODS). >>>>> >>>>> The issue is that I cannot start any virtual machine KVM or container >>>>> LXC; the boot process just hangs after a few seconds. >>>>> All these KVMs and LXCs have in common that their virtual disks >>>>> reside >>>>> in the same pool: hdd >>>>> >>>>> This pool hdd is relatively small compared to the largest pool: >>>>> hdb_backup >>>>> root@ld3955:~# rados df >>>>> POOL_NAME USED OBJECTS CLONES COPIES >>>>> MISSING_ON_PRIMARY >>>>> UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR >>>>> UNDER COMPR >>>>> backup 0 B 0 0 >>>>> 0 >>>>> 0 0 0 0 0 B 0 0 B 0 >>>>> B 0 B >>>>> hdb_backup 589 TiB 51262212 0 >>>>> 153786636 >>>>> 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 >>>>> B 0 B >>>>> hdd 3.2 TiB 281884 6568 >>>>> 845652 >>>>> 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 >>>>> B 0 B >>>>> pve_cephfs_data 955 GiB 91832 0 >>>>> 275496 >>>>> 0 0 3038 2103 1021 MiB 102170 318 GiB 0 >>>>> B 0 B >>>>> pve_cephfs_metadata 486 MiB 62 0 >>>>> 186 >>>>> 0 0 7 860 1.4 GiB 12393 166 MiB 0 >>>>> B 0 B >>>>> >>>>> total_objects 51635990 >>>>> total_used 597 TiB >>>>> total_avail 522 TiB >>>>> total_space 1.1 PiB >>>>> >>>>> This is the current health status of the ceph cluster: >>>>> cluster: >>>>> id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae >>>>> health: HEALTH_ERR >>>>> 1 filesystem is degraded >>>>> 1 MDSs report slow metadata IOs >>>>> 1 backfillfull osd(s) >>>>> 87 nearfull osd(s) >>>>> 1 pool(s) backfillfull >>>>> Reduced data availability: 54 pgs inactive, 47 pgs >>>>> peering, >>>>> 1 pg stale >>>>> Degraded data redundancy: 129598/154907946 objects >>>>> degraded >>>>> (0.084%), 33 pgs degraded, 33 pgs undersized >>>>> Degraded data redundancy (low space): 322 pgs >>>>> backfill_toofull >>>>> 1 subtrees have overcommitted pool target_size_bytes >>>>> 1 subtrees have overcommitted pool target_size_ratio >>>>> 1 pools have too many placement groups >>>>> 21 slow requests are blocked > 32 sec >>>>> >>>>> services: >>>>> mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) >>>>> mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 >>>>> mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby >>>>> osd: 360 osds: 356 up, 356 in; 382 remapped pgs >>>>> >>>>> data: >>>>> pools: 5 pools, 8868 pgs >>>>> objects: 51.64M objects, 197 TiB >>>>> usage: 597 TiB used, 522 TiB / 1.1 PiB avail >>>>> pgs: 0.056% pgs unknown >>>>> 0.553% pgs not active >>>>> 129598/154907946 objects degraded (0.084%) >>>>> 2211119/154907946 objects misplaced (1.427%) >>>>> 8458 active+clean >>>>> 298 active+remapped+backfill_toofull >>>>> 29 remapped+peering >>>>> 24 >>>>> active+undersized+degraded+remapped+backfill_toofull >>>>> 22 active+remapped+backfill_wait >>>>> 17 peering >>>>> 5 unknown >>>>> 5 active+recovery_wait+undersized+degraded+remapped >>>>> 3 active+undersized+degraded+remapped+backfill_wait >>>>> 2 activating+remapped >>>>> 1 active+clean+remapped >>>>> 1 stale+peering >>>>> 1 active+remapped+backfilling >>>>> 1 active+recovering+undersized+remapped >>>>> 1 active+recovery_wait+degraded >>>>> >>>>> io: >>>>> client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr >>>>> >>>>> I believe the cluster is busy with rebalancing pool hdb_backup. >>>>> I set the balance mode upmap recently after the 589TB data was >>>>> written. >>>>> root@ld3955:~# ceph balancer status >>>>> { >>>>> "active": true, >>>>> "plans": [], >>>>> "mode": "upmap" >>>>> } >>>>> >>>>> >>>>> In order to resolve the issue with pool hdd I started some >>>>> investigation. >>>>> First step was to install drivers for the NIC provided Mellanox. >>>>> Then I configured some kernel parameters recommended >>>>> <https://community.mellanox.com/s/article/linux-sysctl-tuning> by >>>>> Mellanox. >>>>> >>>>> However this didn't fix the issue. >>>>> In my opinion I must get rid of all "slow requests are blocked". >>>>> >>>>> When I check the output of ceph health detail any OSD listed under >>>>> REQUEST_SLOW points to an OSD that belongs to pool hdd. >>>>> This means none of the disks belonging to pool hdb_backup is >>>>> showing a >>>>> comparable behaviour. >>>>> >>>>> Then I checked the running processes on the different OSD nodes; I >>>>> use >>>>> tool "glances" here. >>>>> Here I can see single processes that are running for hours and >>>>> consuming >>>>> much CPU, e.g. >>>>> 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S >>>>> 41M 2K >>>>> /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph >>>>> --setgroup ceph >>>>> 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S >>>>> 14M 3K >>>>> /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph >>>>> --setgroup ceph >>>>> >>>>> Similar processes are running on 4 OSD nodes. >>>>> All processes have in common that the relevant OSD belongs to pool >>>>> hdd. >>>>> >>>>> Furthermore glances gives me this alert: >>>>> CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, >>>>> ceph-osd, >>>>> ceph-osd >>>>> >>>>> What can / should I do now? >>>>> Kill the long running processes? >>>>> Stop the relevant OSDs? >>>>> >>>>> Please advise? >>>>> >>>>> THX >>>>> Thomas >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> >>>>> <mailto:ceph-users@ceph.io <mailto:ceph-users@ceph.io>> >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> >>>>> <mailto:ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> >>>> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Hi, I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks. Then I added the 3 OSDs to crushmap. Regards Thomas Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
participants (4)
-
Ashley Merrick
-
Paul Emmerich
-
Thomas
-
Thomas Schneider