On Sat, Sep 21, 2019 at 6:47 PM Thomas <74cmonty@gmail.com> wrote:
Hello,
I have re-created the OSDs using these disks. Can I still export the affected PGs manually?
No, the data probably cannot be recovered in this case :( (It might still be somewhere on the disk if it hasn't been overwritten yet, but it's virtually impossible to recover it: the metadata has almost certainly long been overwritten) But that's only for 17 PGs showing up as unknown. The ones stuck in peering can probably be revived but without the latest writes. Can you run "ceph pg X.YX query" on one of the PGs stuck in peering? It might tell you what's wrong and how to proceed. But even 17 PGs will probably affect almost all of your VMs/containers... Paul
Regards Thomas
Am 20.09.19 um 21:15 schrieb Paul Emmerich:
On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
I cannot get rid of pgs unknown because there were 3 disks that couldn't be started. Therefore I destroyed the relevant OSD and re-created it for the relevant disks. and you had it configured to run with replica 3? Well, I guess the down PGs where located on these three disks that you wiped.
Do you still have the disks? Use ceph-objectstore-tool to export the affected PGs manually and inject them into another OSD.
Paul
Then I added the 3 OSDs to crushmap.
Regards Thomas
Am 20.09.2019 um 08:19 schrieb Ashley Merrick:
Your need to fix this first.
pgs: 0.056% pgs unknown 0.553% pgs not active
The back filling will cause slow I/O, but having pgs unknown and not active will cause I/O blocking which your seeing with the VM booting.
Seems you have 4 OSD's down, if you get them back online you should be able to get all the PG's online.
---- On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas <74cmonty@gmail.com>* wrote ----
Hi,
here I describe 1 of the 2 major issues I'm currently facing in my 8 node ceph cluster (2x MDS, 6x ODS).
The issue is that I cannot start any virtual machine KVM or container LXC; the boot process just hangs after a few seconds. All these KVMs and LXCs have in common that their virtual disks reside in the same pool: hdd
This pool hdd is relatively small compared to the largest pool: hdb_backup root@ld3955:~# rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR backup 0 B 0 0 0 0 0 0 0 0 B 0 0 B 0 B 0 B hdb_backup 589 TiB 51262212 0 153786636 0 0 124895 12266095 4.3 TiB 247132863 463 TiB 0 B 0 B hdd 3.2 TiB 281884 6568 845652 0 0 1658 275277357 16 TiB 208213922 10 TiB 0 B 0 B pve_cephfs_data 955 GiB 91832 0 275496 0 0 3038 2103 1021 MiB 102170 318 GiB 0 B 0 B pve_cephfs_metadata 486 MiB 62 0 186 0 0 7 860 1.4 GiB 12393 166 MiB 0 B 0 B
total_objects 51635990 total_used 597 TiB total_avail 522 TiB total_space 1.1 PiB
This is the current health status of the ceph cluster: cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_ERR 1 filesystem is degraded 1 MDSs report slow metadata IOs 1 backfillfull osd(s) 87 nearfull osd(s) 1 pool(s) backfillfull Reduced data availability: 54 pgs inactive, 47 pgs peering, 1 pg stale Degraded data redundancy: 129598/154907946 objects degraded (0.084%), 33 pgs degraded, 33 pgs undersized Degraded data redundancy (low space): 322 pgs backfill_toofull 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio 1 pools have too many placement groups 21 slow requests are blocked > 32 sec
services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby osd: 360 osds: 356 up, 356 in; 382 remapped pgs
data: pools: 5 pools, 8868 pgs objects: 51.64M objects, 197 TiB usage: 597 TiB used, 522 TiB / 1.1 PiB avail pgs: 0.056% pgs unknown 0.553% pgs not active 129598/154907946 objects degraded (0.084%) 2211119/154907946 objects misplaced (1.427%) 8458 active+clean 298 active+remapped+backfill_toofull 29 remapped+peering 24 active+undersized+degraded+remapped+backfill_toofull 22 active+remapped+backfill_wait 17 peering 5 unknown 5 active+recovery_wait+undersized+degraded+remapped 3 active+undersized+degraded+remapped+backfill_wait 2 activating+remapped 1 active+clean+remapped 1 stale+peering 1 active+remapped+backfilling 1 active+recovering+undersized+remapped 1 active+recovery_wait+degraded
io: client: 9.2 KiB/s wr, 0 op/s rd, 1 op/s wr
I believe the cluster is busy with rebalancing pool hdb_backup. I set the balance mode upmap recently after the 589TB data was written. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" }
In order to resolve the issue with pool hdd I started some investigation. First step was to install drivers for the NIC provided Mellanox. Then I configured some kernel parameters recommended <https://community.mellanox.com/s/article/linux-sysctl-tuning> by Mellanox.
However this didn't fix the issue. In my opinion I must get rid of all "slow requests are blocked".
When I check the output of ceph health detail any OSD listed under REQUEST_SLOW points to an OSD that belongs to pool hdd. This means none of the disks belonging to pool hdb_backup is showing a comparable behaviour.
Then I checked the running processes on the different OSD nodes; I use tool "glances" here. Here I can see single processes that are running for hours and consuming much CPU, e.g. 66.8 0.2 2.13G 1.17G 1192756 ceph 17h8:33 58 0 S 41M 2K /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph 34.2 0.2 4.31G 1.20G 971267 ceph 15h38:46 58 0 S 14M 3K /usr/bin/ceph-osd -f --cluster ceph --id 73 --setuser ceph --setgroup ceph
Similar processes are running on 4 OSD nodes. All processes have in common that the relevant OSD belongs to pool hdd.
Furthermore glances gives me this alert: CRITICAL on CPU_IOWAIT (Min:1.9 Mean:2.3 Max:2.6): ceph-osd, ceph-osd, ceph-osd
What can / should I do now? Kill the long running processes? Stop the relevant OSDs?
Please advise?
THX Thomas _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io