The effect of changing an osd's class
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it. When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd. What happened when I did this?
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD... Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Remapped PGs is exactly what to expect after removing (or adding) a device class. Did you revert the change entirely? It sounds like you maybe forgot to add the original device class back to the OSD where you changed it? Maybe share 'ceph osd tree'? Do you have recovery IO (ceph -s)? Does the number of misplaced objects change? Zitat von Roland Giesler <roland@giesler.za.net>:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2024/11/14 09:37, Eugen Block wrote:
Remapped PGs is exactly what to expect after removing (or adding) a device class. Did you revert the change entirely? It sounds like you maybe forgot to add the original device class back to the OSD where you changed it? Maybe share 'ceph osd tree'? Do you have recovery IO (ceph -s)? Does the number of misplaced objects change?
NodeA:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 49.82324 root default -3 12.36960 host FT1-NodeA 2 hdd 1.86029 osd.2 up 1.00000 1.00000 3 hdd 1.86029 osd.3 up 1.00000 1.00000 4 hdd 1.86029 osd.4 up 1.00000 1.00000 5 hdd 1.86589 osd.5 up 1.00000 1.00000 0 ssd 0.28319 osd.0 up 1.00000 1.00000 1 ssd 0.28319 osd.1 up 1.00000 1.00000 36 ssd 0.20000 osd.36 up 1.00000 1.00000 37 ssd 0.28319 osd.37 up 1.00000 1.00000 38 ssd 0.28319 osd.38 up 1.00000 1.00000 39 ssd 0.28319 osd.39 up 1.00000 1.00000 40 ssd 3.30690 osd.40 up 1.00000 1.00000 -7 12.63338 host FT1-NodeB 10 hdd 1.86029 osd.10 up 1.00000 1.00000 11 hdd 1.86029 osd.11 up 1.00000 1.00000 26 hdd 1.86029 osd.26 up 1.00000 1.00000 27 hdd 1.86029 osd.27 up 1.00000 1.00000 6 ssd 0.28319 osd.6 up 1.00000 1.00000 7 ssd 0.28319 osd.7 up 1.00000 1.00000 8 ssd 0.28319 osd.8 up 1.00000 1.00000 9 ssd 0.28319 osd.9 up 1.00000 1.00000 24 ssd 0.28319 osd.24 up 1.00000 1.00000 25 ssd 0.28319 osd.25 up 1.00000 1.00000 41 ssd 3.49309 osd.41 up 1.00000 1.00000 -10 12.18689 host FT1-NodeC 14 hdd 1.59999 osd.14 up 1.00000 1.00000 15 hdd 1.86029 osd.15 up 1.00000 1.00000 16 hdd 1.86029 osd.16 up 1.00000 1.00000 17 hdd 1.86029 osd.17 up 1.00000 1.00000 12 ssd 0.28319 osd.12 up 1.00000 1.00000 13 ssd 0.28319 osd.13 up 1.00000 1.00000 28 ssd 0.28319 osd.28 up 1.00000 1.00000 29 ssd 0.28319 osd.29 up 1.00000 1.00000 30 ssd 0.28319 osd.30 up 1.00000 1.00000 31 ssd 0.28319 osd.31 up 1.00000 1.00000 43 ssd 3.30690 osd.43 up 1.00000 1.00000 -13 12.63338 host FT1-NodeD 20 hdd 1.86029 osd.20 up 1.00000 1.00000 21 hdd 1.86029 osd.21 up 1.00000 1.00000 22 hdd 1.86029 osd.22 up 1.00000 1.00000 23 hdd 1.86029 osd.23 up 1.00000 1.00000 18 ssd 0.28319 osd.18 up 1.00000 1.00000 19 ssd 0.28319 osd.19 up 1.00000 1.00000 32 ssd 0.28319 osd.32 up 1.00000 1.00000 33 ssd 0.28319 osd.33 up 1.00000 1.00000 34 ssd 0.28319 osd.34 up 1.00000 1.00000 35 ssd 0.28319 osd.35 up 1.00000 1.00000 42 ssd 3.49309 osd.42 up 1.00000 1.00000 No recovery IO, which is what worries me: NodeA:~# ceph -s cluster: id: 04385b88-049f-4083-8d5a-6c45a0b7bddb health: HEALTH_WARN 1 pool(s) have no replicas configured services: mon: 3 daemons, quorum FT1-NodeA,FT1-NodeB,FT1-NodeC (age 3d) mgr: FT1-NodeC(active, since 3d), standbys: FT1-NodeA, FT1-NodeB mds: 1/1 daemons up, 1 standby osd: 44 osds: 44 up (since 14h), 44 in (since 14h); 113 remapped pgs data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11721328 objects misplaced (0.035%) 1471 active+clean 113 active+clean+remapped 2 active+clean+scrubbing+deep io: client: 14 MiB/s rd, 8.5 MiB/s wr, 62 op/s rd, 1.27k op/s wr progress: It stays on that "113 active+clean+remapped". When I "out" osd.0, the status changes for a while (30 minutes estimated) and then, once the recovery IO stops, when I attempt to stop the osd, this is displayed: The 6 "active+remapped+backfilling" pg were many more at first when i took the osd out of the cluster. If I try to stop the osd now however, I see this: Clearly these are more pg's than the 6 that are still backfilling. Is there a way to force the pg's off this osd, so I safely stop it?
Zitat von Roland Giesler <roland@giesler.za.net>:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I had attached images, but these are not shown... On 2024/11/14 10:12, Roland Giesler wrote:
On 2024/11/14 09:37, Eugen Block wrote:
Remapped PGs is exactly what to expect after removing (or adding) a device class. Did you revert the change entirely? It sounds like you maybe forgot to add the original device class back to the OSD where you changed it? Maybe share 'ceph osd tree'? Do you have recovery IO (ceph -s)? Does the number of misplaced objects change?
NodeA:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 49.82324 root default -3 12.36960 host FT1-NodeA 2 hdd 1.86029 osd.2 up 1.00000 1.00000 3 hdd 1.86029 osd.3 up 1.00000 1.00000 4 hdd 1.86029 osd.4 up 1.00000 1.00000 5 hdd 1.86589 osd.5 up 1.00000 1.00000 0 ssd 0.28319 osd.0 up 1.00000 1.00000 1 ssd 0.28319 osd.1 up 1.00000 1.00000 36 ssd 0.20000 osd.36 up 1.00000 1.00000 37 ssd 0.28319 osd.37 up 1.00000 1.00000 38 ssd 0.28319 osd.38 up 1.00000 1.00000 39 ssd 0.28319 osd.39 up 1.00000 1.00000 40 ssd 3.30690 osd.40 up 1.00000 1.00000 -7 12.63338 host FT1-NodeB 10 hdd 1.86029 osd.10 up 1.00000 1.00000 11 hdd 1.86029 osd.11 up 1.00000 1.00000 26 hdd 1.86029 osd.26 up 1.00000 1.00000 27 hdd 1.86029 osd.27 up 1.00000 1.00000 6 ssd 0.28319 osd.6 up 1.00000 1.00000 7 ssd 0.28319 osd.7 up 1.00000 1.00000 8 ssd 0.28319 osd.8 up 1.00000 1.00000 9 ssd 0.28319 osd.9 up 1.00000 1.00000 24 ssd 0.28319 osd.24 up 1.00000 1.00000 25 ssd 0.28319 osd.25 up 1.00000 1.00000 41 ssd 3.49309 osd.41 up 1.00000 1.00000 -10 12.18689 host FT1-NodeC 14 hdd 1.59999 osd.14 up 1.00000 1.00000 15 hdd 1.86029 osd.15 up 1.00000 1.00000 16 hdd 1.86029 osd.16 up 1.00000 1.00000 17 hdd 1.86029 osd.17 up 1.00000 1.00000 12 ssd 0.28319 osd.12 up 1.00000 1.00000 13 ssd 0.28319 osd.13 up 1.00000 1.00000 28 ssd 0.28319 osd.28 up 1.00000 1.00000 29 ssd 0.28319 osd.29 up 1.00000 1.00000 30 ssd 0.28319 osd.30 up 1.00000 1.00000 31 ssd 0.28319 osd.31 up 1.00000 1.00000 43 ssd 3.30690 osd.43 up 1.00000 1.00000 -13 12.63338 host FT1-NodeD 20 hdd 1.86029 osd.20 up 1.00000 1.00000 21 hdd 1.86029 osd.21 up 1.00000 1.00000 22 hdd 1.86029 osd.22 up 1.00000 1.00000 23 hdd 1.86029 osd.23 up 1.00000 1.00000 18 ssd 0.28319 osd.18 up 1.00000 1.00000 19 ssd 0.28319 osd.19 up 1.00000 1.00000 32 ssd 0.28319 osd.32 up 1.00000 1.00000 33 ssd 0.28319 osd.33 up 1.00000 1.00000 34 ssd 0.28319 osd.34 up 1.00000 1.00000 35 ssd 0.28319 osd.35 up 1.00000 1.00000 42 ssd 3.49309 osd.42 up 1.00000 1.00000
No recovery IO, which is what worries me:
NodeA:~# ceph -s cluster: id: 04385b88-049f-4083-8d5a-6c45a0b7bddb health: HEALTH_WARN 1 pool(s) have no replicas configured
services: mon: 3 daemons, quorum FT1-NodeA,FT1-NodeB,FT1-NodeC (age 3d) mgr: FT1-NodeC(active, since 3d), standbys: FT1-NodeA, FT1-NodeB mds: 1/1 daemons up, 1 standby osd: 44 osds: 44 up (since 14h), 44 in (since 14h); 113 remapped pgs
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11721328 objects misplaced (0.035%) 1471 active+clean 113 active+clean+remapped 2 active+clean+scrubbing+deep
io: client: 14 MiB/s rd, 8.5 MiB/s wr, 62 op/s rd, 1.27k op/s wr
progress:
It stays on that "113 active+clean+remapped".
When I "out" osd.0, the status changes for a while (30 minutes estimated) and then, once the recovery IO stops, when I attempt to stop the osd, this is displayed:
Image was here, showing "113 active+clean+remapped", "2 active+clean+scrubbing+deep" and "6 active+remapped+backfilling".
The 6 "active+remapped+backfilling" pg were many more at first when i took the osd out of the cluster. If I try to stop the osd now however, I see this:
Image was here: showing 79 pg's would become offline
Clearly these are more pg's than the 6 that are still backfilling.
Is there a way to force the pg's off this osd, so I safely stop it?
Here's crush rule for these devices: NodeB:/var/log# ceph osd crush rule dump ssd_rule { "rule_id": 0, "rule_name": "ssd_rule", "type": 1, "steps": [ { "op": "take", "item": -2, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }
Zitat von Roland Giesler <roland@giesler.za.net>:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It's not clear to me if you wanted to add some more details after "I see this:" (twice). So you do see backfilling traffic if you out the OSD? Then maybe the remapped PGs are not even on that OSD? Have you checked 'ceph pg ls remapped'? To drain an OSD, you can either set it "out" as you already tried, or set its crush weight to 0: ceph osd crush reweight osd.0 0 Zitat von Roland Giesler <roland@giesler.za.net>:
On 2024/11/14 09:37, Eugen Block wrote:
Remapped PGs is exactly what to expect after removing (or adding) a device class. Did you revert the change entirely? It sounds like you maybe forgot to add the original device class back to the OSD where you changed it? Maybe share 'ceph osd tree'? Do you have recovery IO (ceph -s)? Does the number of misplaced objects change?
NodeA:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 49.82324 root default -3 12.36960 host FT1-NodeA 2 hdd 1.86029 osd.2 up 1.00000 1.00000 3 hdd 1.86029 osd.3 up 1.00000 1.00000 4 hdd 1.86029 osd.4 up 1.00000 1.00000 5 hdd 1.86589 osd.5 up 1.00000 1.00000 0 ssd 0.28319 osd.0 up 1.00000 1.00000 1 ssd 0.28319 osd.1 up 1.00000 1.00000 36 ssd 0.20000 osd.36 up 1.00000 1.00000 37 ssd 0.28319 osd.37 up 1.00000 1.00000 38 ssd 0.28319 osd.38 up 1.00000 1.00000 39 ssd 0.28319 osd.39 up 1.00000 1.00000 40 ssd 3.30690 osd.40 up 1.00000 1.00000 -7 12.63338 host FT1-NodeB 10 hdd 1.86029 osd.10 up 1.00000 1.00000 11 hdd 1.86029 osd.11 up 1.00000 1.00000 26 hdd 1.86029 osd.26 up 1.00000 1.00000 27 hdd 1.86029 osd.27 up 1.00000 1.00000 6 ssd 0.28319 osd.6 up 1.00000 1.00000 7 ssd 0.28319 osd.7 up 1.00000 1.00000 8 ssd 0.28319 osd.8 up 1.00000 1.00000 9 ssd 0.28319 osd.9 up 1.00000 1.00000 24 ssd 0.28319 osd.24 up 1.00000 1.00000 25 ssd 0.28319 osd.25 up 1.00000 1.00000 41 ssd 3.49309 osd.41 up 1.00000 1.00000 -10 12.18689 host FT1-NodeC 14 hdd 1.59999 osd.14 up 1.00000 1.00000 15 hdd 1.86029 osd.15 up 1.00000 1.00000 16 hdd 1.86029 osd.16 up 1.00000 1.00000 17 hdd 1.86029 osd.17 up 1.00000 1.00000 12 ssd 0.28319 osd.12 up 1.00000 1.00000 13 ssd 0.28319 osd.13 up 1.00000 1.00000 28 ssd 0.28319 osd.28 up 1.00000 1.00000 29 ssd 0.28319 osd.29 up 1.00000 1.00000 30 ssd 0.28319 osd.30 up 1.00000 1.00000 31 ssd 0.28319 osd.31 up 1.00000 1.00000 43 ssd 3.30690 osd.43 up 1.00000 1.00000 -13 12.63338 host FT1-NodeD 20 hdd 1.86029 osd.20 up 1.00000 1.00000 21 hdd 1.86029 osd.21 up 1.00000 1.00000 22 hdd 1.86029 osd.22 up 1.00000 1.00000 23 hdd 1.86029 osd.23 up 1.00000 1.00000 18 ssd 0.28319 osd.18 up 1.00000 1.00000 19 ssd 0.28319 osd.19 up 1.00000 1.00000 32 ssd 0.28319 osd.32 up 1.00000 1.00000 33 ssd 0.28319 osd.33 up 1.00000 1.00000 34 ssd 0.28319 osd.34 up 1.00000 1.00000 35 ssd 0.28319 osd.35 up 1.00000 1.00000 42 ssd 3.49309 osd.42 up 1.00000 1.00000
No recovery IO, which is what worries me:
NodeA:~# ceph -s cluster: id: 04385b88-049f-4083-8d5a-6c45a0b7bddb health: HEALTH_WARN 1 pool(s) have no replicas configured
services: mon: 3 daemons, quorum FT1-NodeA,FT1-NodeB,FT1-NodeC (age 3d) mgr: FT1-NodeC(active, since 3d), standbys: FT1-NodeA, FT1-NodeB mds: 1/1 daemons up, 1 standby osd: 44 osds: 44 up (since 14h), 44 in (since 14h); 113 remapped pgs
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11721328 objects misplaced (0.035%) 1471 active+clean 113 active+clean+remapped 2 active+clean+scrubbing+deep
io: client: 14 MiB/s rd, 8.5 MiB/s wr, 62 op/s rd, 1.27k op/s wr
progress:
It stays on that "113 active+clean+remapped".
When I "out" osd.0, the status changes for a while (30 minutes estimated) and then, once the recovery IO stops, when I attempt to stop the osd, this is displayed:
The 6 "active+remapped+backfilling" pg were many more at first when i took the osd out of the cluster. If I try to stop the osd now however, I see this:
Clearly these are more pg's than the 6 that are still backfilling.
Is there a way to force the pg's off this osd, so I safely stop it?
Zitat von Roland Giesler <roland@giesler.za.net>:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I know the similar behaviour when mclock is active. Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Roland Giesler <roland@giesler.za.net> schrieb am Do., 14. Nov. 2024, 05:40:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote:
I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2024/11/14 11:44, Joachim Kraftmayer wrote:
I know the similar behaviour when mclock is active.
For osd.0 I see: osd.0 basic osd_mclock_max_capacity_iops_ssd 14305.161403 I'm unfamiliar with mclock. Can one tune that to improve the situation? Roland
Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Roland Giesler <roland@giesler.za.net> schrieb am Do., 14. Nov. 2024, 05:40:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote: I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it. When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd. What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Roland, Yes, you can. See mclock documentation here [1]. One think I can think of is that these 113 PGs may have a common misbehaving OSD (primary or not) with a ridiculous osd_mclock_max_capacity_iops_ssd value set. Restarting the primary and/or adjusting osd_mclock_max_capacity_iops_ssd value(s) could help in this situation. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/ ----- Le 14 Nov 24, à 12:19, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/14 11:44, Joachim Kraftmayer wrote:
I know the similar behaviour when mclock is active.
For osd.0 I see:
osd.0 basic osd_mclock_max_capacity_iops_ssd 14305.161403
I'm unfamiliar with mclock. Can one tune that to improve the situation?
Roland
Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Roland Giesler <roland@giesler.za.net> schrieb am Do., 14. Nov. 2024, 05:40:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote: I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it. When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd. What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I redid what I did before. I changed the osd class and had a look at the storage and found a VM image! I deleted it (it was a copy) and now the storage is empty. I guess the image (assigned to a non-existent storage after reverting left pg's that could not be moved. I'm reverting now again to see if I can remove the osd's. On 2024/11/14 15:22, Frédéric Nass wrote:
Hi Roland,
Yes, you can. See mclock documentation here [1].
One think I can think of is that these 113 PGs may have a common misbehaving OSD (primary or not) with a ridiculous osd_mclock_max_capacity_iops_ssd value set. Restarting the primary and/or adjusting osd_mclock_max_capacity_iops_ssd value(s) could help in this situation.
Regards, Frédéric.
[1] https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/
----- Le 14 Nov 24, à 12:19, Roland Giesler roland@giesler.za.net a écrit :
On 2024/11/14 11:44, Joachim Kraftmayer wrote:
I know the similar behaviour when mclock is active. For osd.0 I see:
osd.0 basic osd_mclock_max_capacity_iops_ssd 14305.161403
I'm unfamiliar with mclock. Can one tune that to improve the situation?
Roland
Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Roland Giesler <roland@giesler.za.net> schrieb am Do., 14. Nov. 2024, 05:40:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that it all went back when you reverted. I would not expect a mess.
data: volumes: 1/1 healthy pools: 7 pools, 1586 pgs objects: 5.79M objects, 12 TiB usage: 24 TiB used, 26 TiB / 50 TiB avail pgs: 4161/11720662 objects misplaced (0.036%) 1463 active+clean 113 active+clean+remapped 9 active+clean+scrubbing+deep+repair 1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If I try to out the this osd to stop it, the clsuter also never settles. So then if I try to stop it in the GUI is tells me 73 pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler <roland@giesler.za.net> wrote: I created a new osd class and changed the class of an osd to the new one without taking the osd out and stopping it first. The new class also has a crush rule and a pool created for it. When I realised my mistake, I reverted to what I had before. However, I suspect that I now have a mess on that osd. What happened when I did this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You might also first try ceph osd down 1701 This marks the OSD down in the map, it doesn’t restart anything, but it does serve in some cases to goose progress. The OSD will quickly mark itself back up. Where 1701 is the ID of said primary. ceph health detail is one way that you might see the acting sets of involved PGs, the primary will be the first ID in each’s list. Chances are that you’ll see a common OSD in the sets.
On Nov 14, 2024, at 8:22 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
One think I can think of is that these 113 PGs may have a common misbehaving OSD (primary or not) with a ridiculous osd_mclock_max_capacity_iops_ssd value set. Restarting the primary and/or adjusting osd_mclock_max_capacity_iops_ssd value(s) could help in this situation.
How do I determine the primary osd? On 2024/11/14 16:12, Anthony D'Atri wrote:
You might also first try
ceph osd down 1701
This marks the OSD down in the map, it doesn’t restart anything, but it does serve in some cases to goose progress. The OSD will quickly mark itself back up.
Where 1701 is the ID of said primary.
ceph health detail
is one way that you might see the acting sets of involved PGs, the primary will be the first ID in each’s list. Chances are that you’ll see a common OSD in the sets.
On Nov 14, 2024, at 8:22 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
One think I can think of is that these 113 PGs may have a common misbehaving OSD (primary or not) with a ridiculous osd_mclock_max_capacity_iops_ssd value set. Restarting the primary and/or adjusting osd_mclock_max_capacity_iops_ssd value(s) could help in this situation.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 2024/11/15 13:00, Gregory Orange wrote:
On 15/11/24 17:11, Roland Giesler wrote:
How do I determine the primary osd? ceph pg map $pg
ceph pg $pg query | jq .info.stats.acting_primary
You can jq and less to take a look at other values which might be informative too.
Ah, of course :-) Sorry, I was looking for the primary osd of a pool! No wonder I couldn't find anything. LOL!
Greg. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
All the problem pg's are on osd.39. When I stop osd.39, it shows 86 pg's would be offline. However, there is no recovery that happens. It just stays there. 86 undersized+remapped+peered I managed to pin down all the pg groups that are in this state by using: ceph pg dump | grep active+clean+remapped From there I queried the first pg on the list: NodeC:~# ceph pg 28.42 query { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "active+clean+remapped", "epoch": 490547, "up": [], "acting": [ 39, 1 ], "acting_recovery_backfill": [ "1", "39" ], "info": { "pgid": "28.42", "last_update": "489784'67", "last_complete": "489784'67", "log_tail": "0'0", "last_user_version": 67, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 487438, "epoch_pool_created": 487438, "last_epoch_started": 490544, "last_interval_started": 490543, "last_epoch_clean": 490544, "last_interval_clean": 490543, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 489999, "same_interval_since": 490543, "same_primary_since": 490543, "last_scrub": "489784'67", "last_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2024-11-12T16:55:34.967587+0200", "last_clean_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "prior_readable_until_ub": 0 }, "stats": { "version": "489784'67", "reported_seq": 721, "reported_epoch": 490547, "state": "active+clean+remapped", "last_fresh": "2024-11-15T16:07:08.111992+0200", "last_change": "2024-11-15T15:01:12.081136+0200", "last_active": "2024-11-15T16:07:08.111992+0200", "last_peered": "2024-11-15T16:07:08.111992+0200", "last_clean": "2024-11-15T16:07:08.111992+0200", "last_became_active": "2024-11-15T15:01:12.080691+0200", "last_became_peered": "2024-11-15T15:01:12.080691+0200", "last_unstale": "2024-11-15T16:07:08.111992+0200", "last_undegraded": "2024-11-15T16:07:08.111992+0200", "last_fullsized": "2024-11-15T16:07:08.111992+0200", "mapping_epoch": 490543, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 487438, "last_epoch_clean": 490544, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "489784'67", "last_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2024-11-12T16:55:34.967587+0200", "last_clean_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "objects_scrubbed": 0, "log_size": 67, "ondisk_log_size": 67, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "last_scrub_duration": 1, "scrub_schedule": "periodic scrub scheduled @ 2024-11-16T17:19:20.646231+0000", "scrub_duration": 0.024065349, "objects_trimmed": 0, "snaptrim_duration": 0.090716250999999998, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 68, "num_read_kb": 708, "num_write": 67, "num_write_kb": 19088, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 12, "num_bytes_recovered": 26607616, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [ 39, 1 ], "avail_no_missing": [ "39", "1" ], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": 39, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 490544, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "1", "pgid": "28.42", "last_update": "489784'67", "last_complete": "489784'67", "log_tail": "0'0", "last_user_version": 67, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 487438, "epoch_pool_created": 487438, "last_epoch_started": 490544, "last_interval_started": 490543, "last_epoch_clean": 490544, "last_interval_clean": 490543, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 489999, "same_interval_since": 490543, "same_primary_since": 490543, "last_scrub": "489784'67", "last_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2024-11-12T16:55:34.967587+0200", "last_clean_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "prior_readable_until_ub": 0 }, "stats": { "version": "489784'67", "reported_seq": 430, "reported_epoch": 490542, "state": "undersized+remapped+peered", "last_fresh": "2024-11-15T15:01:10.046703+0200", "last_change": "2024-11-15T15:00:37.072082+0200", "last_active": "2024-11-14T15:41:06.074260+0200", "last_peered": "2024-11-15T15:01:10.046703+0200", "last_clean": "2024-11-14T15:41:03.945754+0200", "last_became_active": "2024-11-14T15:39:15.997707+0200", "last_became_peered": "2024-11-15T15:00:37.072082+0200", "last_unstale": "2024-11-15T15:01:10.046703+0200", "last_undegraded": "2024-11-15T15:01:10.046703+0200", "last_fullsized": "2024-11-15T15:00:37.069250+0200", "mapping_epoch": 490543, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 487438, "last_epoch_clean": 490528, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "489784'67", "last_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "2024-11-12T16:55:34.967587+0200", "last_clean_scrub_stamp": "2024-11-15T09:16:59.715671+0200", "objects_scrubbed": 0, "log_size": 67, "ondisk_log_size": 67, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "last_scrub_duration": 0, "scrub_schedule": "periodic scrub scheduled @ 2024-11-16T12:15:01.587848+0000", "scrub_duration": 0, "objects_trimmed": 0, "snaptrim_duration": 0.090716250999999998, "stat_sum": { "num_bytes": 0, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 68, "num_read_kb": 708, "num_write": 67, "num_write_kb": 19088, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 7, "num_bytes_recovered": 26607616, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [], "acting": [ 39, 1 ], "avail_no_missing": [ "1" ], "object_location_counts": [], "blocked_by": [], "up_primary": -1, "acting_primary": 39, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 490544, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2024-11-15T15:01:12.073557+0200", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [], "waiting_on_backfill": [], "last_backfill_started": "MIN", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "pull_from_peer": [], "pushing": [] } } }, { "name": "Started", "enter_time": "2024-11-15T15:01:11.069697+0200" } ], "scrubber": { "active": false, "must_scrub": false, "must_deep_scrub": false, "must_repair": false, "need_auto": false, "scrub_reg_stamp": "2024-11-16T19:19:20.646231+0200", "schedule": "scrub scheduled @ 2024-11-16T17:19:20.646231+0000" }, "agent_state": {} } Can anyone see from this what the reason may be that this (and the other pg's) are stuck on this osd? On 2024/11/15 13:36, Roland Giesler wrote:
On 2024/11/15 13:00, Gregory Orange wrote:
On 15/11/24 17:11, Roland Giesler wrote:
How do I determine the primary osd? ceph pg map $pg
ceph pg $pg query | jq .info.stats.acting_primary
You can jq and less to take a look at other values which might be informative too.
Ah, of course :-) Sorry, I was looking for the primary osd of a pool! No wonder I couldn't find anything. LOL!
Greg. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Anthony D'Atri
-
Eugen Block
-
Frédéric Nass
-
Gregory Orange
-
Joachim Kraftmayer
-
Roland Giesler