Problems with autoscaler (overlapping roots) after changing the pool class
Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html
Hi, what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore. Regards Eugen Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in the following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas how to fix the problem.
I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I tried the following on a small testbed first: ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore Thanks a lot ! Cheers, Massimo On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in the following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas how to fix the problem.
I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I got a similar problem after changing pool class to use only hdd following https://www.spinics.net/lists/ceph-users/msg84987.html. Data migrated successfully. I get warnings like: 2024-12-23T14:39:37.103+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.index won't scale due to overlapping roots: {-1, -18} 2024-12-23T14:39:37.105+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.data won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.107+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.111+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling ... while crush tree with shadow shows: -2 hdd 1043.93188 root default~hdd -4 hdd 151.82336 host ctplosd1~hdd 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... -1 1050.48230 root default -3 153.27872 host ctplosd1 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... and even though crush rule for example for pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw is set to: { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, and I still get warning messages. Is there a way I can check if a particular "root" is used somewhere other than go thorough ceph osd pool ls detail and look into crush rule? Can I somehow delete "old" root default? Would it be safe to change pg manually even with overlapped roots? Rok On Wed, Jan 25, 2023 at 12:03 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I tried the following on a small testbed first:
ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd
and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore
Thanks a lot !
Cheers, Massimo
On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in the following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas how to fix the problem.
I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Don't try to delete a root, that will definitely break something. Instead, check the crush rules which don't use a device class and use the reclassify of the crushtool to modify the rules. This will trigger only a bit of data movement, but not as much as a simple change of the rule would. Zitat von Rok Jaklič <rjaklic@gmail.com>:
I got a similar problem after changing pool class to use only hdd following https://www.spinics.net/lists/ceph-users/msg84987.html. Data migrated successfully.
I get warnings like: 2024-12-23T14:39:37.103+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.index won't scale due to overlapping roots: {-1, -18} 2024-12-23T14:39:37.105+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.data won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.107+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.111+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling ...
while crush tree with shadow shows: -2 hdd 1043.93188 root default~hdd -4 hdd 151.82336 host ctplosd1~hdd 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... -1 1050.48230 root default -3 153.27872 host ctplosd1 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ...
and even though crush rule for example for
pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw
is set to: { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] },
and I still get warning messages.
Is there a way I can check if a particular "root" is used somewhere other than go thorough ceph osd pool ls detail and look into crush rule?
Can I somehow delete "old" root default?
Would it be safe to change pg manually even with overlapped roots?
Rok
On Wed, Jan 25, 2023 at 12:03 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I tried the following on a small testbed first:
ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd
and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore
Thanks a lot !
Cheers, Massimo
On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in the following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas how to fix the problem.
I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That would be then replicated_rule [root@ctplmon1 ~]# ceph osd pool ls detail pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 320144 flags hashpspool stripe_width 0 pg_num_min 1 application mgr,mgr_devicehealth pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 320144 lfor 0/18964/18962 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 320144 lfor 0/127672/127670 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 320144 lfor 0/59850/59848 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 320144 lfor 0/51538/51536 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 6 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule 2 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 315285 lfor 0/127830/127828 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw pool 7 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 320144 lfor 0/76474/76472 flags hashpspool stripe_width 0 application rgw pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw pool 10 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 320144 flags hashpspool,bulk stripe_width 0 application cephfs pool 11 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 4 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 320144 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs [root@ctplmon1 ~]# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, I would go then adding "class ssd" after "step take default" in crushmap just like in https://www.spinics.net/lists/ceph-users/msg84987.html # rules rule replicated_rule { id 0 type replicated step take default class ssd step chooseleaf firstn 0 type host step emit } --- I expect that data will move from hdd to ssd. Will services be unavailable during movement? Or is it maybe safe also just to set pools to another rule: { "rule_id": 2, "rule_name": "replicated_ssd", "type": 1, "steps": [ { "op": "take", "item": -18, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, is it maybe another option just to reset pool crush_rule e.g.: ceph osd pool set .mgr crush_rule replicated_ssd ? Rok On Mon, Dec 23, 2024 at 3:12 PM Eugen Block <eblock@nde.ag> wrote:
Don't try to delete a root, that will definitely break something. Instead, check the crush rules which don't use a device class and use the reclassify of the crushtool to modify the rules. This will trigger only a bit of data movement, but not as much as a simple change of the rule would.
Zitat von Rok Jaklič <rjaklic@gmail.com>:
I got a similar problem after changing pool class to use only hdd following https://www.spinics.net/lists/ceph-users/msg84987.html. Data migrated successfully.
I get warnings like: 2024-12-23T14:39:37.103+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.index won't scale due to overlapping roots: {-1, -18} 2024-12-23T14:39:37.105+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.data won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.107+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.111+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling ...
while crush tree with shadow shows: -2 hdd 1043.93188 root default~hdd -4 hdd 151.82336 host ctplosd1~hdd 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... -1 1050.48230 root default -3 153.27872 host ctplosd1 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ...
and even though crush rule for example for
pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw
is set to: { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] },
and I still get warning messages.
Is there a way I can check if a particular "root" is used somewhere other than go thorough ceph osd pool ls detail and look into crush rule?
Can I somehow delete "old" root default?
Would it be safe to change pg manually even with overlapped roots?
Rok
On Wed, Jan 25, 2023 at 12:03 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I tried the following on a small testbed first:
ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd
and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore
Thanks a lot !
Cheers, Massimo
On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in the following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas how to fix the problem.
I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Agreed. The .mgr pool is a usual suspect here, especially when using Rook. When any pool is constrained to a device class, this kind of warning will happen if *all* pools don’t specify one. Of course there’s also the strategy of disabling the autoscaler, but that takes more analysis. We old farts are used to it, but it can be daunting for whippersnappers.
On Dec 23, 2024, at 9:11 AM, Eugen Block <eblock@nde.ag> wrote:
Don't try to delete a root, that will definitely break something. Instead, check the crush rules which don't use a device class and use the reclassify of the crushtool to modify the rules. This will trigger only a bit of data movement, but not as much as a simple change of the rule would.
Zitat von Rok Jaklič <rjaklic@gmail.com>:
I got a similar problem after changing pool class to use only hdd following https://www.spinics.net/lists/ceph-users/msg84987.html. Data migrated successfully.
I get warnings like: 2024-12-23T14:39:37.103+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.index won't scale due to overlapping roots: {-1, -18} 2024-12-23T14:39:37.105+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.data won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.107+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.111+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling ...
while crush tree with shadow shows: -2 hdd 1043.93188 root default~hdd -4 hdd 151.82336 host ctplosd1~hdd 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... -1 1050.48230 root default -3 153.27872 host ctplosd1 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ...
and even though crush rule for example for
pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw
is set to: { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] },
and I still get warning messages.
Is there a way I can check if a particular "root" is used somewhere other than go thorough ceph osd pool ls detail and look into crush rule?
Can I somehow delete "old" root default?
Would it be safe to change pg manually even with overlapped roots?
Rok
On Wed, Jan 25, 2023 at 12:03 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I tried the following on a small testbed first:
ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd
and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore
Thanks a lot !
Cheers, Massimo
On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in the following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas how to fix the problem.
I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I will try changing/adding class ssd to replicated_rule tomorrow even though I am a little hesitant for some reason to edit this rule since it could mean that system data for rgw would "stay somewhere" if something goes wrong. I was much braver when I was changing the rule for EC32 where I separated OSD data to just hdd, since "some data" was already on hdd. On Mon, Dec 23, 2024 at 4:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Agreed. The .mgr pool is a usual suspect here, especially when using Rook. When any pool is constrained to a device class, this kind of warning will happen if *all* pools don’t specify one.
Of course there’s also the strategy of disabling the autoscaler, but that takes more analysis. We old farts are used to it, but it can be daunting for whippersnappers.
On Dec 23, 2024, at 9:11 AM, Eugen Block <eblock@nde.ag> wrote:
Don't try to delete a root, that will definitely break something. Instead, check the crush rules which don't use a device class and use the reclassify of the crushtool to modify the rules. This will trigger only a bit of data movement, but not as much as a simple change of the rule would.
Zitat von Rok Jaklič <rjaklic@gmail.com>:
I got a similar problem after changing pool class to use only hdd following https://www.spinics.net/lists/ceph-users/msg84987.html. Data migrated successfully.
I get warnings like: 2024-12-23T14:39:37.103+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.index won't scale due to overlapping roots: {-1, -18} 2024-12-23T14:39:37.105+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.data won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.107+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.111+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling ...
while crush tree with shadow shows: -2 hdd 1043.93188 root default~hdd -4 hdd 151.82336 host ctplosd1~hdd 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... -1 1050.48230 root default -3 153.27872 host ctplosd1 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ...
and even though crush rule for example for
pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw
is set to: { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] },
and I still get warning messages.
Is there a way I can check if a particular "root" is used somewhere other than go thorough ceph osd pool ls detail and look into crush rule?
Can I somehow delete "old" root default?
Would it be safe to change pg manually even with overlapped roots?
Rok
On Wed, Jan 25, 2023 at 12:03 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I tried the following on a small testbed first:
ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd
and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore
Thanks a lot !
Cheers, Massimo
On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in the following way:
ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas how to fix the problem.
I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools)
Any suggestions ?
The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If your NVMe OSDs have the `ssd` device class, doing what you suggest might not even result in any data movement. https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/#migrating-fr... This page shows how to use the reclassify feature to help avoid typos when editing the CRUSHmap. Using a CLI tool when feasible makes this sort of thing a lot safer, compared to back in the day when we had to text-edit everything by hand :nailbiting:. One can readily diff the before and after decompiled text CRUSHmaps to ensure sanity before recompiling and injecting. I’ve done this myself multiple times since device classes became a thing.
On Dec 23, 2024, at 5:05 PM, Rok Jaklič <rjaklic@gmail.com> wrote:
I will try changing/adding class ssd to replicated_rule tomorrow even though I am a little hesitant for some reason to edit this rule since it could mean that system data for rgw would "stay somewhere" if something goes wrong. I was much braver when I was changing the rule for EC32 where I separated OSD data to just hdd, since "some data" was already on hdd.
On Mon, Dec 23, 2024 at 4:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Agreed. The .mgr pool is a usual suspect here, especially when using Rook. When any pool is constrained to a device class, this kind of warning will happen if *all* pools don’t specify one.
Of course there’s also the strategy of disabling the autoscaler, but that takes more analysis. We old farts are used to it, but it can be daunting for whippersnappers.
On Dec 23, 2024, at 9:11 AM, Eugen Block <eblock@nde.ag> wrote:
Don't try to delete a root, that will definitely break something. Instead, check the crush rules which don't use a device class and use the reclassify of the crushtool to modify the rules. This will trigger only a bit of data movement, but not as much as a simple change of the rule would.
Zitat von Rok Jaklič <rjaklic@gmail.com>:
I got a similar problem after changing pool class to use only hdd following https://www.spinics.net/lists/ceph-users/msg84987.html. Data migrated successfully.
I get warnings like: 2024-12-23T14:39:37.103+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.index won't scale due to overlapping roots: {-1, -18} 2024-12-23T14:39:37.105+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool default.rgw.buckets.data won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.107+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.111+0100 7f949edad640 0 [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling ...
while crush tree with shadow shows: -2 hdd 1043.93188 root default~hdd -4 hdd 151.82336 host ctplosd1~hdd 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... -1 1050.48230 root default -3 153.27872 host ctplosd1 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ...
and even though crush rule for example for
pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw
is set to: { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] },
and I still get warning messages.
Is there a way I can check if a particular "root" is used somewhere other than go thorough ceph osd pool ls detail and look into crush rule?
Can I somehow delete "old" root default?
Would it be safe to change pg manually even with overlapped roots?
Rok
On Wed, Jan 25, 2023 at 12:03 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I tried the following on a small testbed first:
ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd
and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore
Thanks a lot !
Cheers, Massimo
On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
> Dear all > > I have just changed the crush rule for all the replicated pools in the > following way: > > ceph osd crush rule create-replicated replicated_hdd default host hdd > ceph osd pool set <poolname> crush_rule replicated_hdd > > See also this [*] thread > Before applying this change, these pools were all using > the replicated_ruleset rule where the class is not specified. > > > > I am noticing now a problem with the autoscaler: "ceph osd pool > autoscale-status" doesn't report any output and the mgr log complains about > overlapping roots: > > [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} > > > Indeed: > > # ceph osd crush tree --show-shadow > ID CLASS WEIGHT TYPE NAME > -18 hdd 1329.26501 root default~hdd > -17 hdd 329.14154 rack Rack11-PianoAlto~hdd > -15 hdd 54.56085 host ceph-osd-04~hdd > 30 hdd 5.45609 osd.30 > 31 hdd 5.45609 osd.31 > ... > ... > -1 1329.26501 root default > -7 329.14154 rack Rack11-PianoAlto > -8 54.56085 host ceph-osd-04 > 30 hdd 5.45609 osd.30 > 31 hdd 5.45609 osd.31 > ... > > I have already read about this behavior but I have no clear ideas how to > fix the problem. > > I read somewhere that the problem happens when there are rules that force > some pools to only use one class and there are also pools which does not > make any distinction between device classes > > > All the replicated pools are using the replicated_hdd pool but I also have > some EC pools which are using a profile where the class is not specified. > As far I understand, I can't force these pools to use only the hdd class: > according to the doc I can't change this profile specifying the hdd class > (or at least the change wouldn't be applied to the existing EC pools) > > Any suggestions ? > > The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if > you want to have a look > > Many thanks, Massimo > > [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I followed then Eugens advice with something like: - create a new rule via CLI which includes a device class - dump the crushmap again and test the new rule with crushtool - If the output is as expected, assign the new rule to a pool of your choice, I'd start with a less important one. - If everything's good, do the same for all necessary pools, wait for remapping to finish. - No pool should be using the default "replicated_rule" now. - Dump a fresh crushmap and decompile it - add a "class hdd" entry to the default replicated_rule - save and compile - inject the modified crushmap (with this single change), nothing should happen in the cluster since no pool should use the replicated_rule at that point. ... After I've changed pools to a new rule, remapping started, warning messages do not show any more and PGs for pools are increasing. Thank you all for the help. Rok On Tue, Dec 24, 2024 at 2:16 AM Anthony D'Atri <anthony.datri@gmail.com> wrote:
If your NVMe OSDs have the `ssd` device class, doing what you suggest might not even result in any data movement.
docs.ceph.com <https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/#migrating-from-a-legacy-ssd-rule-to-device-classes>
<https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/#migrating-from-a-legacy-ssd-rule-to-device-classes> <https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/#migrating-from-a-legacy-ssd-rule-to-device-classes>
This page shows how to use the reclassify feature to help avoid typos when editing the CRUSHmap. Using a CLI tool when feasible makes this sort of thing a lot safer, compared to back in the day when we had to text-edit everything by hand :nailbiting:. One can readily diff the before and after decompiled text CRUSHmaps to ensure sanity before recompiling and injecting.
I’ve done this myself multiple times since device classes became a thing.
On Dec 23, 2024, at 5:05 PM, Rok Jaklič <rjaklic@gmail.com> wrote:
I will try changing/adding class ssd to replicated_rule tomorrow even though I am a little hesitant for some reason to edit this rule since it could mean that system data for rgw would "stay somewhere" if something goes wrong. I was much braver when I was changing the rule for EC32 where I separated OSD data to just hdd, since "some data" was already on hdd.
On Mon, Dec 23, 2024 at 4:12 PM Anthony D'Atri <anthony.datri@gmail.com> wrote:
Agreed. The .mgr pool is a usual suspect here, especially when using Rook. When any pool is constrained to a device class, this kind of warning will happen if *all* pools don’t specify one.
Of course there’s also the strategy of disabling the autoscaler, but that takes more analysis. We old farts are used to it, but it can be daunting for whippersnappers.
On Dec 23, 2024, at 9:11 AM, Eugen Block <eblock@nde.ag> wrote:
Don't try to delete a root, that will definitely break something.
Instead, check the crush rules which don't use a device class and use the reclassify of the crushtool to modify the rules. This will trigger only a bit of data movement, but not as much as a simple change of the rule would.
Zitat von Rok Jaklič <rjaklic@gmail.com>:
I got a similar problem after changing pool class to use only hdd
following
https://www.spinics.net/lists/ceph-users/msg84987.html. Data migrated successfully.
I get warnings like: 2024-12-23T14:39:37.103+0100 7f949edad640 0 [pg_autoscaler WARNING
root]
pool default.rgw.buckets.index won't scale due to overlapping roots:
{-1,
-18} 2024-12-23T14:39:37.105+0100 7f949edad640 0 [pg_autoscaler WARNING
root]
pool default.rgw.buckets.data won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.107+0100 7f949edad640 0 [pg_autoscaler WARNING
root]
pool cephfs_metadata won't scale due to overlapping roots: {-2, -1, -18} 2024-12-23T14:39:37.111+0100 7f949edad640 0 [pg_autoscaler WARNING
root]
pool 1 contains an overlapping root -1... skipping scaling ...
while crush tree with shadow shows: -2 hdd 1043.93188 root default~hdd -4 hdd 151.82336 host ctplosd1~hdd 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ... -1 1050.48230 root default -3 153.27872 host ctplosd1 0 hdd 5.45798 osd.0 1 hdd 5.45798 osd.1 2 hdd 5.45798 osd.2 3 hdd 5.45798 osd.3 4 hdd 5.45798 osd.4 ...
and even though crush rule for example for
pool 9 'default.rgw.buckets.data' erasure profile ec-32-profile size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 320144 lfor 0/127784/214408 flags hashpspool,ec_overwrites stripe_width 12288 application rgw
is set to: { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] },
and I still get warning messages.
Is there a way I can check if a particular "root" is used somewhere
other
than go thorough ceph osd pool ls detail and look into crush rule?
Can I somehow delete "old" root default?
Would it be safe to change pg manually even with overlapped roots?
Rok
On Wed, Jan 25, 2023 at 12:03 PM Massimo Sgaravatto < massimo.sgaravatto@gmail.com> wrote:
I tried the following on a small testbed first:
ceph osd erasure-code-profile set profile-4-2-hdd k=4 m=2 crush-failure-domain=host crush-device-class=hdd ceph osd crush rule create-erasure ecrule-4-2-hdd profile-4-2-hdd ceph osd pool set ecpool-4-2 crush_rule ecrule-4-2-hdd
and indeed after having applied this change for all the EC pools, the autoscaler doesn't complain anymore
Thanks a lot !
Cheers, Massimo
On Tue, Jan 24, 2023 at 7:02 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
what you can’t change with EC pools is the EC profile, the pool‘s ruleset you can change. The fix is the same as for the replicates pools, assign a ruleset with hdd class and after some data movement the autoscaler should not complain anymore.
Regards Eugen
Zitat von Massimo Sgaravatto <massimo.sgaravatto@gmail.com>:
Dear all
I have just changed the crush rule for all the replicated pools in
the
following way:
ceph osd crush rule create-replicated replicated_hdd default host
hdd
ceph osd pool set <poolname> crush_rule replicated_hdd
See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified.
I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log
complains
about
overlapping roots:
[pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18,
-1}
Indeed:
# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ...
I have already read about this behavior but I have no clear ideas
how
to
fix the problem.
I read somewhere that the problem happens when there are rules that
force
some pools to only use one class and there are also pools which
does
not
make any distinction between device classes
All the replicated pools are using the replicated_hdd pool but I
also
have
some EC pools which are using a profile where the class is not
specified.
As far I understand, I can't force these pools to use only the hdd
class:
according to the doc I can't change this profile specifying the hdd
class
(or at least the change wouldn't be applied to the existing EC
pools)
Any suggestions ?
The crush map is available at
https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr,
if
you want to have a look
Many thanks, Massimo
[*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Eugen Block
-
Massimo Sgaravatto
-
Rok Jaklič