Overlapping Roots - How to Fix?
Helllo, I've reviewed some recent posts in this list and also searched Google for info about autoscale and overlapping roots. In what I have found I do not see anything that I can understand regarding how to fix the issue - probably because I don't deal with Crush on a regular basis. My particulars: I have an up-to-date Reef cluster in production with some EC pools and some replicated pools. My error messages look like: pool default.rgw.buckets.data won't scale due to overlapping roots: {-1, -2} From what I read and looking at 'ceph osd crush rule dump', it looks like the 8 replicated pools have "op": "take", "item": -1, "item_name": "default" whereas the 2 EC pools have "op": "take", "item": -2, "item_name": "default~hdd" To be sure, all of my OSDs are identical - HDD with SSD WAL/DB. Please advise on how to fix this. Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu
Helllo,
I've reviewed some recent posts in this list and also searched Google for info about autoscale and overlapping roots. In what I have found I do not see anything that I can understand regarding how to fix the issue - probably because I don't deal with Crush on a regular basis.
Checkout the Note in this section: https://docs.ceph.com/en/reef/rados/operations/placement-groups/#viewing-pg-... I added that last year I think it was as a result of how Rook was creating pools.
From what I read and looking at 'ceph osd crush rule dump', it looks like the 8 replicated pools have
"op": "take", "item": -1, "item_name": "default"
whereas the 2 EC pools have
"op": "take", "item": -2, "item_name": "default~hdd"
To be sure, all of my OSDs are identical - HDD with SSD WAL/DB.
Please advise on how to fix this.
The subtlety that's easy to miss is that when you specify a device class for only *some* pools, the pools/rules that specify a device class effectively act on a "shadow" CRUSH root. My terminology may be inexact there. So I think if you adjust your CRUSH rules so that they all specify a device class -- in your case all the same device class -- your problem (and balancer performance perhaps) will improve.
Anthony, So it sounds like I need to make a new crush rule for replicated pools that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.) Then, after I create this new rule, I simply assign the pool to a new crush rule using a command similar to the one shown in your note in the link you referenced? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Wed, Sep 18, 2024 at 2:10 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Helllo,
I've reviewed some recent posts in this list and also searched Google for info about autoscale and overlapping roots. In what I have found I do not see anything that I can understand regarding how to fix the issue - probably because I don't deal with Crush on a regular basis.
Checkout the Note in this section: https://docs.ceph.com/en/reef/rados/operations/placement-groups/#viewing-pg-...
I added that last year I think it was as a result of how Rook was creating pools.
From what I read and looking at 'ceph osd crush rule dump', it looks like the 8 replicated pools have
"op": "take", "item": -1, "item_name": "default"
whereas the 2 EC pools have
"op": "take", "item": -2, "item_name": "default~hdd"
To be sure, all of my OSDs are identical - HDD with SSD WAL/DB.
Please advise on how to fix this.
The subtlety that's easy to miss is that when you specify a device class for only *some* pools, the pools/rules that specify a device class effectively act on a "shadow" CRUSH root. My terminology may be inexact there.
So I think if you adjust your CRUSH rules so that they all specify a device class -- in your case all the same device class -- your problem (and balancer performance perhaps) will improve.
Anthony,
So it sounds like I need to make a new crush rule for replicated pools that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.)
I think it would be best to edit the CRUSH rules in-situ so that each specifies the device class, that way if you do get different media in the future, you'll be ready. Rather than messing around with new rules and modifying pools, this is arguably one of the few times when one would decompile, edit, recompile, and inject the CRUSH map in toto. I haven't tried this myself, but maybe something like the below, to avoid the PITA and potential for error of edting the decompiled text file by hand. ceph osd getcrushmap -o original.crush crushtool -d original.crush -o original.txt crushtool -i original.crush --reclassify --reclassify-root default hdd --set-subtree-class default hdd -o adjusted.crush crushtool -d adjusted.crush -o adjusted.txt crushtool -i original.crush --compare adjusted.crush ceph osd setcrushmap -i adjusted.crush
Then, after I create this new rule, I simply assign the pool to a new crush rule using a command similar to the one shown in your note in the link you referenced?
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu <mailto:kdhall@binghamton.edu>
On Wed, Sep 18, 2024 at 2:10 PM Anthony D'Atri <anthony.datri@gmail.com <mailto:anthony.datri@gmail.com>> wrote:
Helllo,
I've reviewed some recent posts in this list and also searched Google for info about autoscale and overlapping roots. In what I have found I do not see anything that I can understand regarding how to fix the issue - probably because I don't deal with Crush on a regular basis.
Checkout the Note in this section: https://docs.ceph.com/en/reef/rados/operations/placement-groups/#viewing-pg-...
I added that last year I think it was as a result of how Rook was creating pools.
From what I read and looking at 'ceph osd crush rule dump', it looks like the 8 replicated pools have
"op": "take", "item": -1, "item_name": "default"
whereas the 2 EC pools have
"op": "take", "item": -2, "item_name": "default~hdd"
To be sure, all of my OSDs are identical - HDD with SSD WAL/DB.
Please advise on how to fix this.
The subtlety that's easy to miss is that when you specify a device class for only *some* pools, the pools/rules that specify a device class effectively act on a "shadow" CRUSH root. My terminology may be inexact there.
So I think if you adjust your CRUSH rules so that they all specify a device class -- in your case all the same device class -- your problem (and balancer performance perhaps) will improve.
On 19-09-2024 05:10, Anthony D'Atri wrote:
Anthony,
So it sounds like I need to make a new crush rule for replicated pools that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.)
I think it would be best to edit the CRUSH rules in-situ so that each specifies the device class, that way if you do get different media in the future, you'll be ready. Rather than messing around with new rules and modifying pools, this is arguably one of the few times when one would decompile, edit, recompile, and inject the CRUSH map in toto.
I haven't tried this myself, but maybe something like the below, to avoid the PITA and potential for error of edting the decompiled text file by hand.
ceph osd getcrushmap -o original.crush crushtool -d original.crush -o original.txt crushtool -i original.crush --reclassify --reclassify-root default hdd --set-subtree-class default hdd -o adjusted.crush crushtool -d adjusted.crush -o adjusted.txt crushtool -i original.crush --compare adjusted.crush ceph osd setcrushmap -i adjusted.crush
This might be of use as well (if a lot of data would move): https://blog.widodh.nl/2019/02/comparing-two-ceph-crush-maps/ Gr. Stefan
Stefan, Anthony, Anthony's sequence of commands to reclassify the root failed with errors. so I have tried to look a little deeper. I can now see the extra root via 'ceph osd crush tree --show-shadow'. Looking at the decompiled crush tree, I can also see the extra root: root default { id -1 # do not change unnecessarily * id -2 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 } Based on the hints given in the link provided by Stefan, it would appear that the correct solution might be to get rid of 'id -2' and change id -1 to class hdd, root default { * id -1 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 } but I'm no expert and anxious about losing data. The rest of the rules in my crush map are: # rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule block-1 { id 1 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule default.rgw.buckets.data { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule ceph-block { id 3 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule replicated-hdd { id 4 type replicated step take default class hdd step choose firstn 0 type osd step emit } # end crush map Of these, the last - id 4 - is one that I added while trying to figure this out. What this tells me is that the 'take' step in rule id 0 should probably change to 'step take default class hdd'. I also notice that each of my host stanzas (buckets) has what looks like two roots. For example host ceph00 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 90.51434 alg straw2 hash 0 # rjenkins1 item osd.0 weight 11.35069 item osd.1 weight 11.35069 item osd.2 weight 11.35069 item osd.3 weight 11.35069 item osd.4 weight 11.27789 item osd.5 weight 11.27789 item osd.6 weight 11.27789 item osd.7 weight 11.27789 } I assume I may need to clean this up somehow, or perhaps this is the real problem. Please advise. Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Thu, Sep 19, 2024 at 3:56 AM Stefan Kooman <stefan@bit.nl> wrote:
On 19-09-2024 05:10, Anthony D'Atri wrote:
Anthony,
So it sounds like I need to make a new crush rule for replicated pools
that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.)
I think it would be best to edit the CRUSH rules in-situ so that each
specifies the device class, that way if you do get different media in the future, you'll be ready. Rather than messing around with new rules and modifying pools, this is arguably one of the few times when one would decompile, edit, recompile, and inject the CRUSH map in toto.
I haven't tried this myself, but maybe something like the below, to
avoid the PITA and potential for error of edting the decompiled text file by hand.
ceph osd getcrushmap -o original.crush crushtool -d original.crush -o original.txt crushtool -i original.crush --reclassify --reclassify-root default hdd
--set-subtree-class default hdd -o adjusted.crush
crushtool -d adjusted.crush -o adjusted.txt crushtool -i original.crush --compare adjusted.crush ceph osd setcrushmap -i adjusted.crush
This might be of use as well (if a lot of data would move): https://blog.widodh.nl/2019/02/comparing-two-ceph-crush-maps/
Gr. Stefan
Well, it was pasted from a local cluster, meant as a guide not to be run literally.
On Sep 20, 2024, at 12:48 PM, Dave Hall <kdhall@binghamton.edu> wrote:
Stefan, Anthony,
Anthony's sequence of commands to reclassify the root failed with errors. so I have tried to look a little deeper.
I can now see the extra root via 'ceph osd crush tree --show-shadow'. Looking at the decompiled crush tree, I can also see the extra root:
root default { id -1 # do not change unnecessarily id -2 class hdd # do not change unnecessarily # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
Based on the hints given in the link provided by Stefan, it would appear that the correct solution might be to get rid of 'id -2' and change id -1 to class hdd,
root default { id -1 class hdd # do not change unnecessarily # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
but I'm no expert and anxious about losing data.
The rest of the rules in my crush map are:
# rules rule replicated_rule { id 0 type replicated step take default # missing device class step chooseleaf firstn 0 type host step emit } rule block-1 { id 1 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule default.rgw.buckets.data { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule ceph-block { id 3 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule replicated-hdd { id 4 type replicated step take default class hdd step choose firstn 0 type osd step emit }
# end crush map
Of these, the last - id 4 - is one that I added while trying to figure this out. What this tells me is that the 'take' step in rule id 0 should probably change to 'step take default class hdd'.
I also notice that each of my host stanzas (buckets) has what looks like two roots. For example
host ceph00 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 90.51434 alg straw2 hash 0 # rjenkins1 item osd.0 weight 11.35069 item osd.1 weight 11.35069 item osd.2 weight 11.35069 item osd.3 weight 11.35069 item osd.4 weight 11.27789 item osd.5 weight 11.27789 item osd.6 weight 11.27789 item osd.7 weight 11.27789 }
I assume I may need to clean this up somehow, or perhaps this is the real problem.
Please advise.
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu <mailto:kdhall@binghamton.edu>
On Thu, Sep 19, 2024 at 3:56 AM Stefan Kooman <stefan@bit.nl <mailto:stefan@bit.nl>> wrote:
On 19-09-2024 05:10, Anthony D'Atri wrote:
Anthony,
So it sounds like I need to make a new crush rule for replicated pools that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.)
I think it would be best to edit the CRUSH rules in-situ so that each specifies the device class, that way if you do get different media in the future, you'll be ready. Rather than messing around with new rules and modifying pools, this is arguably one of the few times when one would decompile, edit, recompile, and inject the CRUSH map in toto.
I haven't tried this myself, but maybe something like the below, to avoid the PITA and potential for error of edting the decompiled text file by hand.
ceph osd getcrushmap -o original.crush crushtool -d original.crush -o original.txt crushtool -i original.crush --reclassify --reclassify-root default hdd --set-subtree-class default hdd -o adjusted.crush crushtool -d adjusted.crush -o adjusted.txt crushtool -i original.crush --compare adjusted.crush ceph osd setcrushmap -i adjusted.crush
This might be of use as well (if a lot of data would move): https://blog.widodh.nl/2019/02/comparing-two-ceph-crush-maps/
Gr. Stefan
Oddly, the Nautilus cluster that I'm gradually decommissioning seems to have the same shadow root pattern in its crush map. I don't know if that really means anything, but at least I know it's not something I did differently when I set up the new Reef cluster. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Fri, Sep 20, 2024 at 12:48 PM Dave Hall <kdhall@binghamton.edu> wrote:
Stefan, Anthony,
Anthony's sequence of commands to reclassify the root failed with errors. so I have tried to look a little deeper.
I can now see the extra root via 'ceph osd crush tree --show-shadow'. Looking at the decompiled crush tree, I can also see the extra root:
root default { id -1 # do not change unnecessarily
* id -2 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
Based on the hints given in the link provided by Stefan, it would appear that the correct solution might be to get rid of 'id -2' and change id -1 to class hdd,
root default {
* id -1 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
but I'm no expert and anxious about losing data.
The rest of the rules in my crush map are:
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule block-1 { id 1 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule default.rgw.buckets.data { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule ceph-block { id 3 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule replicated-hdd { id 4 type replicated step take default class hdd step choose firstn 0 type osd step emit }
# end crush map
Of these, the last - id 4 - is one that I added while trying to figure this out. What this tells me is that the 'take' step in rule id 0 should probably change to 'step take default class hdd'.
I also notice that each of my host stanzas (buckets) has what looks like two roots. For example
host ceph00 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 90.51434 alg straw2 hash 0 # rjenkins1 item osd.0 weight 11.35069 item osd.1 weight 11.35069 item osd.2 weight 11.35069 item osd.3 weight 11.35069 item osd.4 weight 11.27789 item osd.5 weight 11.27789 item osd.6 weight 11.27789 item osd.7 weight 11.27789 }
I assume I may need to clean this up somehow, or perhaps this is the real problem.
Please advise.
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Thu, Sep 19, 2024 at 3:56 AM Stefan Kooman <stefan@bit.nl> wrote:
On 19-09-2024 05:10, Anthony D'Atri wrote:
Anthony,
So it sounds like I need to make a new crush rule for replicated pools
that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.)
I think it would be best to edit the CRUSH rules in-situ so that each
specifies the device class, that way if you do get different media in the future, you'll be ready. Rather than messing around with new rules and modifying pools, this is arguably one of the few times when one would decompile, edit, recompile, and inject the CRUSH map in toto.
I haven't tried this myself, but maybe something like the below, to
avoid the PITA and potential for error of edting the decompiled text file by hand.
ceph osd getcrushmap -o original.crush crushtool -d original.crush -o original.txt crushtool -i original.crush --reclassify --reclassify-root default hdd
--set-subtree-class default hdd -o adjusted.crush
crushtool -d adjusted.crush -o adjusted.txt crushtool -i original.crush --compare adjusted.crush ceph osd setcrushmap -i adjusted.crush
This might be of use as well (if a lot of data would move): https://blog.widodh.nl/2019/02/comparing-two-ceph-crush-maps/
Gr. Stefan
I think it would suffice to change rule 0 to use a device class as well, as you already mentioned yourself. Do you have pools that use that rule? If not, the change wouldn’t even have any impact. Zitat von Dave Hall <kdhall@binghamton.edu>:
Oddly, the Nautilus cluster that I'm gradually decommissioning seems to have the same shadow root pattern in its crush map. I don't know if that really means anything, but at least I know it's not something I did differently when I set up the new Reef cluster.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Fri, Sep 20, 2024 at 12:48 PM Dave Hall <kdhall@binghamton.edu> wrote:
Stefan, Anthony,
Anthony's sequence of commands to reclassify the root failed with errors. so I have tried to look a little deeper.
I can now see the extra root via 'ceph osd crush tree --show-shadow'. Looking at the decompiled crush tree, I can also see the extra root:
root default { id -1 # do not change unnecessarily
* id -2 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
Based on the hints given in the link provided by Stefan, it would appear that the correct solution might be to get rid of 'id -2' and change id -1 to class hdd,
root default {
* id -1 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
but I'm no expert and anxious about losing data.
The rest of the rules in my crush map are:
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule block-1 { id 1 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule default.rgw.buckets.data { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule ceph-block { id 3 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule replicated-hdd { id 4 type replicated step take default class hdd step choose firstn 0 type osd step emit }
# end crush map
Of these, the last - id 4 - is one that I added while trying to figure this out. What this tells me is that the 'take' step in rule id 0 should probably change to 'step take default class hdd'.
I also notice that each of my host stanzas (buckets) has what looks like two roots. For example
host ceph00 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 90.51434 alg straw2 hash 0 # rjenkins1 item osd.0 weight 11.35069 item osd.1 weight 11.35069 item osd.2 weight 11.35069 item osd.3 weight 11.35069 item osd.4 weight 11.27789 item osd.5 weight 11.27789 item osd.6 weight 11.27789 item osd.7 weight 11.27789 }
I assume I may need to clean this up somehow, or perhaps this is the real problem.
Please advise.
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Thu, Sep 19, 2024 at 3:56 AM Stefan Kooman <stefan@bit.nl> wrote:
On 19-09-2024 05:10, Anthony D'Atri wrote:
Anthony,
So it sounds like I need to make a new crush rule for replicated pools
that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.)
I think it would be best to edit the CRUSH rules in-situ so that each
specifies the device class, that way if you do get different media in the future, you'll be ready. Rather than messing around with new rules and modifying pools, this is arguably one of the few times when one would decompile, edit, recompile, and inject the CRUSH map in toto.
I haven't tried this myself, but maybe something like the below, to
avoid the PITA and potential for error of edting the decompiled text file by hand.
ceph osd getcrushmap -o original.crush crushtool -d original.crush -o original.txt crushtool -i original.crush --reclassify --reclassify-root default hdd
--set-subtree-class default hdd -o adjusted.crush
crushtool -d adjusted.crush -o adjusted.txt crushtool -i original.crush --compare adjusted.crush ceph osd setcrushmap -i adjusted.crush
This might be of use as well (if a lot of data would move): https://blog.widodh.nl/2019/02/comparing-two-ceph-crush-maps/
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you to everybody who has responded to my questions. At this point I think I am starting to understand. However, I am still trying to understand the potential for data loss. In particular: - In some ways it seems that as long as there is sufficient OSD capacity available the worst that can happen from a bad crush map is poor placement and poor performance. Is this correct? - crushtool --compare - if the result of this command shows no mismatches, can we say that the adjusted crush map is safe to apply? - If all of the 'inhibit flags' are turned on (no out, no down, no scrub/deep-scrub, no recover/rebalance/backfill, and perhaps pause) is it safe to apply an adjusted crush map? Is it safe to revert to the original crush map if things don't seem quite right? Thanks. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu On Sat, Sep 21, 2024 at 4:01 AM Eugen Block <eblock@nde.ag> wrote:
I think it would suffice to change rule 0 to use a device class as well, as you already mentioned yourself. Do you have pools that use that rule? If not, the change wouldn’t even have any impact.
Zitat von Dave Hall <kdhall@binghamton.edu>:
Oddly, the Nautilus cluster that I'm gradually decommissioning seems to have the same shadow root pattern in its crush map. I don't know if that really means anything, but at least I know it's not something I did differently when I set up the new Reef cluster.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Fri, Sep 20, 2024 at 12:48 PM Dave Hall <kdhall@binghamton.edu> wrote:
Stefan, Anthony,
Anthony's sequence of commands to reclassify the root failed with errors. so I have tried to look a little deeper.
I can now see the extra root via 'ceph osd crush tree --show-shadow'. Looking at the decompiled crush tree, I can also see the extra root:
root default { id -1 # do not change unnecessarily
* id -2 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
Based on the hints given in the link provided by Stefan, it would appear that the correct solution might be to get rid of 'id -2' and change id -1 to class hdd,
root default {
* id -1 class hdd # do not change unnecessarily* # weight 361.90518 alg straw2 hash 0 # rjenkins1 item ceph00 weight 90.51434 item ceph01 weight 90.29265 item ceph09 weight 90.80554 item ceph02 weight 90.29265 }
but I'm no expert and anxious about losing data.
The rest of the rules in my crush map are:
# rules rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type host step emit } rule block-1 { id 1 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule default.rgw.buckets.data { id 2 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule ceph-block { id 3 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type osd step emit } rule replicated-hdd { id 4 type replicated step take default class hdd step choose firstn 0 type osd step emit }
# end crush map
Of these, the last - id 4 - is one that I added while trying to figure this out. What this tells me is that the 'take' step in rule id 0 should probably change to 'step take default class hdd'.
I also notice that each of my host stanzas (buckets) has what looks like two roots. For example
host ceph00 { id -3 # do not change unnecessarily id -4 class hdd # do not change unnecessarily # weight 90.51434 alg straw2 hash 0 # rjenkins1 item osd.0 weight 11.35069 item osd.1 weight 11.35069 item osd.2 weight 11.35069 item osd.3 weight 11.35069 item osd.4 weight 11.27789 item osd.5 weight 11.27789 item osd.6 weight 11.27789 item osd.7 weight 11.27789 }
I assume I may need to clean this up somehow, or perhaps this is the real problem.
Please advise.
Thanks.
-Dave
-- Dave Hall Binghamton University kdhall@binghamton.edu
On Thu, Sep 19, 2024 at 3:56 AM Stefan Kooman <stefan@bit.nl> wrote:
On 19-09-2024 05:10, Anthony D'Atri wrote:
Anthony,
So it sounds like I need to make a new crush rule for replicated
pools that specifies default-hdd and the device class? (Or should I go the other way around? I think I'd rather change the replicated pools even though there's more of them.)
I think it would be best to edit the CRUSH rules in-situ so that each
specifies the device class, that way if you do get different media in the future, you'll be ready. Rather than messing around with new rules and modifying pools, this is arguably one of the few times when one would decompile, edit, recompile, and inject the CRUSH map in toto.
I haven't tried this myself, but maybe something like the below, to
avoid the PITA and potential for error of edting the decompiled text file by hand.
ceph osd getcrushmap -o original.crush crushtool -d original.crush -o original.txt crushtool -i original.crush --reclassify --reclassify-root default
hdd --set-subtree-class default hdd -o adjusted.crush
crushtool -d adjusted.crush -o adjusted.txt crushtool -i original.crush --compare adjusted.crush ceph osd setcrushmap -i adjusted.crush
This might be of use as well (if a lot of data would move): https://blog.widodh.nl/2019/02/comparing-two-ceph-crush-maps/
Gr. Stefan
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 23-09-2024 16:04, Dave Hall wrote:
Thank you to everybody who has responded to my questions.
At this point I think I am starting to understand. However, I am still trying to understand the potential for data loss.
In particular:
- In some ways it seems that as long as there is sufficient OSD capacity available the worst that can happen from a bad crush map is poor placement and poor performance. Is this correct?
If you would have a (new) crush rule without any OSD mappings all PGs for pools that use that rule would go in an inactive state, i.e. downtime. So when you create a (new) rule you would have to check that CRUSH can indeed find enough OSDs to comply with the policy you defined.
- crushtool --compare - if the result of this command shows no mismatches, can we say that the adjusted crush map is safe to apply?
Do you mean there are no differences? Then yes.
- If all of the 'inhibit flags' are turned on (no out, no down, no scrub/deep-scrub, no recover/rebalance/backfill, and perhaps pause) is it safe to apply an adjusted crush map?
It should always be "safe". It just depends on the kind of CRUSHmap you are injecting (crush rules without any valid mappings) that might "break" things. Ceph will handle that and inform you about it, but you might run into downtime when "wrong" CRUSHmap is injected. So you should always be careful and test the CRUSHmap beforehand. Is it safe to revert to the original
crush map if things don't seem quite right?
Yes. That's _exactly_ the think you should do when things do not go according to plan. So make sure you have a working CRUSHmap (backup) that you can fall back to (ceph osd getcrushmap -o /tmp/crush_raw). Gr. Stefan
Den mån 23 sep. 2024 kl 16:23 skrev Stefan Kooman <stefan@bit.nl>:
On 23-09-2024 16:04, Dave Hall wrote:
Thank you to everybody who has responded to my questions.
At this point I think I am starting to understand. However, I am still trying to understand the potential for data loss.
In particular:
- In some ways it seems that as long as there is sufficient OSD capacity available the worst that can happen from a bad crush map is poor placement and poor performance. Is this correct?
If you would have a (new) crush rule without any OSD mappings all PGs for pools that use that rule would go in an inactive state, i.e. downtime. So when you create a (new) rule you would have to check that CRUSH can indeed find enough OSDs to comply with the policy you defined.
Are you sure? I have asked some pools to use an "impossible" crush rule after creation and the PGs only end up as "misplaced". At creation they might stay inactive until a good place for them can be found, but then you can't write data to it so it is not a "data-loss" scenario really if the pool never started.
On 23-09-2024 16:31, Janne Johansson wrote:
Den mån 23 sep. 2024 kl 16:23 skrev Stefan Kooman <stefan@bit.nl>:
On 23-09-2024 16:04, Dave Hall wrote:
Thank you to everybody who has responded to my questions.
At this point I think I am starting to understand. However, I am still trying to understand the potential for data loss.
In particular:
- In some ways it seems that as long as there is sufficient OSD capacity available the worst that can happen from a bad crush map is poor placement and poor performance. Is this correct?
If you would have a (new) crush rule without any OSD mappings all PGs for pools that use that rule would go in an inactive state, i.e. downtime. So when you create a (new) rule you would have to check that CRUSH can indeed find enough OSDs to comply with the policy you defined.
Are you sure? I have asked some pools to use an "impossible" crush rule after creation and the PGs only end up as "misplaced".
Apparently it depends ... You are right with regard to the newly created pools and inactive state (at least that's what I have seen in all cases). If I have a pool use a crush rule that does not have any OSDs (nvme class rule without OSDs with nvme device class) the PGs become "unknown" (not state inactive like I said). But IO for that pool does not work at that point (both acting and up sets have "[]p-1", i.e. no OSDs available). However, if I switch back to a valid rule, and then back again to the invalid rule, the PGs become "active+clean+remapped" and IO does work (up set is: []p-1 but acting set is previously mapped OSDs). That is probably the same state you have seen in your cluster (and I have seen in Reef clusters as well). My tests have been performed on a 16.2.11 test cluster. At
creation they might stay inactive until a good place for them can be found, but then you can't write data to it so it is not a "data-loss" scenario really if the pool never started.
Correct, no data loss in that situation. Gr. Stefan
participants (5)
-
Anthony D'Atri
-
Dave Hall
-
Eugen Block
-
Janne Johansson
-
Stefan Kooman