On 2024/11/14 09:37, Eugen Block wrote:
Remapped PGs is exactly what to expect after removing (or adding) a device class. Did you revert the change entirely? It sounds like you maybe forgot to add the original device class back to the OSD where you changed it? Maybe share 'ceph osd tree'? Do you have recovery IO (ceph -s)? Does the number of misplaced objects change?
NodeA:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 49.82324 root default -3 12.36960 host FT1-NodeA 2 hdd 1.86029 osd.2 up 1.00000 1.00000 3 hdd 1.86029 osd.3 up 1.00000 1.00000 4 hdd 1.86029 osd.4 up 1.00000 1.00000 5 hdd 1.86589 osd.5 up 1.00000 1.00000 0 ssd 0.28319 osd.0 up 1.00000 1.00000 1 ssd 0.28319 osd.1 up 1.00000 1.00000 36 ssd 0.20000 osd.36 up 1.00000 1.00000 37 ssd 0.28319 osd.37 up 1.00000 1.00000 38 ssd 0.28319 osd.38 up 1.00000 1.00000 39 ssd 0.28319 osd.39 up 1.00000 1.00000 40 ssd 3.30690 osd.40 up 1.00000 1.00000 -7 12.63338 host FT1-NodeB 10 hdd 1.86029 osd.10 up 1.00000 1.00000 11 hdd 1.86029 osd.11 up 1.00000 1.00000 26 hdd 1.86029 osd.26 up 1.00000 1.00000 27 hdd 1.86029 osd.27 up 1.00000 1.00000 6 ssd 0.28319 osd.6 up 1.00000 1.00000 7 ssd 0.28319 osd.7 up 1.00000 1.00000 8 ssd 0.28319 osd.8 up 1.00000 1.00000 9 ssd 0.28319 osd.9 up 1.00000 1.00000 24 ssd 0.28319 osd.24 up 1.00000 1.00000 25 ssd 0.28319 osd.25 up 1.00000 1.00000 41 ssd 3.49309 osd.41 up 1.00000 1.00000 -10 12.18689 host FT1-NodeC 14 hdd 1.59999 osd.14 up 1.00000 1.00000 15 hdd 1.86029 osd.15 up 1.00000 1.00000 16 hdd 1.86029 osd.16 up 1.00000 1.00000 17 hdd 1.86029 osd.17 up 1.00000 1.00000 12 ssd 0.28319 osd.12 up 1.00000 1.00000 13 ssd 0.28319 osd.13 up 1.00000 1.00000 28 ssd 0.28319 osd.28 up 1.00000 1.00000 29 ssd 0.28319 osd.29 up 1.00000 1.00000 30 ssd 0.28319 osd.30 up 1.00000 1.00000 31 ssd 0.28319 osd.31 up 1.00000 1.00000 43 ssd 3.30690 osd.43 up 1.00000 1.00000 -13 12.63338 host FT1-NodeD 20 hdd 1.86029 osd.20 up 1.00000 1.00000 21 hdd 1.86029 osd.21 up 1.00000 1.00000 22 hdd 1.86029 osd.22 up 1.00000 1.00000 23 hdd 1.86029 osd.23 up 1.00000 1.00000 18 ssd 0.28319 osd.18 up 1.00000 1.00000 19 ssd 0.28319 osd.19 up 1.00000 1.00000 32 ssd 0.28319 osd.32 up 1.00000 1.00000 33 ssd 0.28319 osd.33 up 1.00000 1.00000 34 ssd 0.28319 osd.34 up 1.00000 1.00000 35 ssd 0.28319 osd.35 up 1.00000 1.00000 42 ssd 3.49309 osd.42 up 1.00000 1.00000 No recovery IO, which is what worries me: NodeA:~# ceph -s cluster: id: 04385b88-049f-4083-8d5a-6c45a0b7bddb health: HEALTH_WARN 1 pool(s) have no replicas configured services: mon: 3 daemons, quorum FT1-NodeA,FT1-NodeB,FT1-NodeC (age 3d) mgr: FT1-NodeC(active, since 3d), standbys: FT1-NodeA, FT1-NodeB mds: 1/1 daemons up, 1 standby osd: 44 osds: 44 up (since 14h), 44 in (since 14h); 113 remapped pgs data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11721328 objects misplaced (0.035%) 1471 active+clean 113 active+clean+remapped 2 active+clean+scrubbing+deep io: client: 14 MiB/s rd, 8.5 MiB/s wr, 62 op/s rd, 1.27k op/s wr progress: It stays on that "113 active+clean+remapped". When I "out" osd.0, the status changes for a while (30 minutes estimated) and then, once the recovery IO stops, when I attempt to stop the osd, this is displayed: The 6 "active+remapped+backfilling" pg were many more at first when i took the osd out of the cluster. If I try to stop the osd now however, I see this: Clearly these are more pg's than the 6 that are still backfilling. Is there a way to force the pg's off this osd, so I safely stop it?
Zitat von Roland Giesler <roland@giesler.za.net>:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io