Quincy: osd_pool_default_crush_rule being ignored?
Hello everyone, my cluster has two CRUSH rules: the default replicated_rule (rule_id 0), and another rule named rack-aware (rule_id 1). Now, if I'm not misreading the config reference, I should be able to define that all future-created pools use the rack-aware rule, by setting osd_pool_default_crush_rule to 1. I've verified that this option is defined in src/common/options/global.yaml.in, so the "global" configuration section should be the applicable one (I did try with "mon" and "osd" also, for good measure). However, setting this option, in Quincy, apparently has no effect: # ceph config set global osd_pool_default_crush_rule 1 # ceph osd pool create foo pool 'foo' created # ceph osd pool ls detail | grep foo # pool 9 'foo' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 264 flags hashpspool stripe_width 0 I am seeing this behaviour in 17.2.7. After an upgrade to Reef (18.2.4) it is gone, the option behaves as documented, and new pools are created with a crush_rule of 1: # ceph osd pool create bar pool 'bar' created # ceph osd pool ls detail | grep bar pool 10 'bar' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 302 flags hashpspool stripe_width 0 read_balance_score 4.00 However, the documentation at https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#conf... asserts that osd_pool_default_crush_rule should already work in Quincy, and the Reef release notes at https://docs.ceph.com/en/latest/releases/reef/ don't mention a fix covering this. Am I doing something wrong? Is this a documentation bug, and the option can't work in Quincy? Was this "accidentally" fixed at some point in the Reef cycle? Thanks in advance for any insight you might be able to share. Cheers, Florian
Hi, for me this worked in a 17.2.7 cluster just fine (except for erasure-coded pools). quincy-1:~ # ceph osd crush rule create-replicated new-rule default osd hdd quincy-1:~ # ceph config set mon osd_pool_default_crush_rule 1 quincy-1:~ # ceph osd pool create test-pool2 pool 'test-pool2' created quincy-1:~ # ceph osd pool ls detail | grep test-pool2 pool 20 'test-pool2' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 2593 flags hashpspool stripe_width 0 quincy-1:~ # ceph versions { ... "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 11 } } Setting the option globally works as well for me. Regards, Eugen Zitat von Florian Haas <florian@citynetwork.eu>:
Hello everyone,
my cluster has two CRUSH rules: the default replicated_rule (rule_id 0), and another rule named rack-aware (rule_id 1).
Now, if I'm not misreading the config reference, I should be able to define that all future-created pools use the rack-aware rule, by setting osd_pool_default_crush_rule to 1.
I've verified that this option is defined in src/common/options/global.yaml.in, so the "global" configuration section should be the applicable one (I did try with "mon" and "osd" also, for good measure).
However, setting this option, in Quincy, apparently has no effect:
# ceph config set global osd_pool_default_crush_rule 1 # ceph osd pool create foo pool 'foo' created # ceph osd pool ls detail | grep foo # pool 9 'foo' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 264 flags hashpspool stripe_width 0
I am seeing this behaviour in 17.2.7. After an upgrade to Reef (18.2.4) it is gone, the option behaves as documented, and new pools are created with a crush_rule of 1:
# ceph osd pool create bar pool 'bar' created # ceph osd pool ls detail | grep bar pool 10 'bar' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 302 flags hashpspool stripe_width 0 read_balance_score 4.00
However, the documentation at https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#conf... asserts that osd_pool_default_crush_rule should already work in Quincy, and the Reef release notes at https://docs.ceph.com/en/latest/releases/reef/ don't mention a fix covering this.
Am I doing something wrong? Is this a documentation bug, and the option can't work in Quincy? Was this "accidentally" fixed at some point in the Reef cycle?
Thanks in advance for any insight you might be able to share.
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 25/09/2024 09:05, Eugen Block wrote:
Hi,
for me this worked in a 17.2.7 cluster just fine
Huh, interesting!
(except for erasure-coded pools).
Okay, *that* bit is expected. https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#conf... does say that the option sets the "default CRUSH rule to use when creating a replicated pool".
quincy-1:~ # ceph osd crush rule create-replicated new-rule default osd hdd
Mine was a rule created with "create-simple"; would that make a difference? Cheers, Florian
On 25/09/2024 09:05, Eugen Block wrote:
Hi,
for me this worked in a 17.2.7 cluster just fine
Huh, interesting!
(except for erasure-coded pools).
Okay, *that* bit is expected. https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#conf... does say that the option sets the "default CRUSH rule to use when creating a replicated pool".
quincy-1:~ # ceph osd crush rule create-replicated new-rule default osd hdd
Mine was a rule created with "create-simple"; would that make a difference? Cheers, Florian
Still works: quincy-1:~ # ceph osd crush rule create-simple simple-rule default osd quincy-1:~ # ceph osd crush rule dump simple-rule { "rule_id": 4, ... quincy-1:~ # ceph config set mon osd_pool_default_crush_rule 4 quincy-1:~ # ceph osd pool create test-pool6 pool 'test-pool6' created quincy-1:~ # ceph osd pool ls detail | grep test-pool pool 24 'test-pool6' replicated size 2 min_size 1 crush_rule 4 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 2615 flags hashpspool stripe_width 0 Zitat von Florian Haas <florian@citynetwork.eu>:
On 25/09/2024 09:05, Eugen Block wrote:
Hi,
for me this worked in a 17.2.7 cluster just fine
Huh, interesting!
(except for erasure-coded pools).
Okay, *that* bit is expected. https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#conf... does say that the option sets the "default CRUSH rule to use when creating a replicated pool".
quincy-1:~ # ceph osd crush rule create-replicated new-rule default osd hdd
Mine was a rule created with "create-simple"; would that make a difference?
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, I've just torn down and completely respun my cluster, on 17.2.7. Recreated my CRUSH rule, set osd_pool_default_crush_rule to its rule_id, 1. Created a new pool. That new pool still has crush_rule 0, just as before and contrary to what you're seeing. I'm a bit puzzled, because I'm out of ideas as to what could break on my cluster and work fine on yours, to cause this. Odd. Cheers, Florian
Hm, do you have any local ceph.conf on your client which has an override for this option as well? By the way, how do you bootstrap your cluster? Is it cephadm based? Zitat von Florian Haas <florian@citynetwork.eu>:
Hi Eugen,
I've just torn down and completely respun my cluster, on 17.2.7.
Recreated my CRUSH rule, set osd_pool_default_crush_rule to its rule_id, 1.
Created a new pool.
That new pool still has crush_rule 0, just as before and contrary to what you're seeing.
I'm a bit puzzled, because I'm out of ideas as to what could break on my cluster and work fine on yours, to cause this. Odd.
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I redeployed a different single-node-cluster with quincy 17.2.6 and it works there as well. Zitat von Eugen Block <eblock@nde.ag>:
Hm, do you have any local ceph.conf on your client which has an override for this option as well? By the way, how do you bootstrap your cluster? Is it cephadm based?
Zitat von Florian Haas <florian@citynetwork.eu>:
Hi Eugen,
I've just torn down and completely respun my cluster, on 17.2.7.
Recreated my CRUSH rule, set osd_pool_default_crush_rule to its rule_id, 1.
Created a new pool.
That new pool still has crush_rule 0, just as before and contrary to what you're seeing.
I'm a bit puzzled, because I'm out of ideas as to what could break on my cluster and work fine on yours, to cause this. Odd.
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 25/09/2024 15:21, Eugen Block wrote:
Hm, do you have any local ceph.conf on your client which has an override for this option as well?
No.
By the way, how do you bootstrap your cluster? Is it cephadm based?
This one is bootstrapped (on Quincy) with ceph-ansible. And when the "ceph config set" change didn't make a difference, I did also make a point of cycling all my mons and osds (which shouldn't be necessary, but I figured I'd try that, just in case). And I also confirmed this same issue, in Quincy, after the cluster was adopted into cephadm management. At that point, the behaviour was still unchanged. It was only after I upgraded the cluster to Reef, with cephadm/ceph orch, that the problem went away. Cheers, Florian
Hm, I don't know much about ceph-ansible. Did you check if there was any config set for a specific daemon or something, which would override the config set? For example, 'ceph config show-with-defaults mon.<MON>' for each mon, and then also check 'ceph config dump | grep rule'. I would also probably grep for crush_rule in all the usual places. Zitat von Florian Haas <florian@citynetwork.eu>:
On 25/09/2024 15:21, Eugen Block wrote:
Hm, do you have any local ceph.conf on your client which has an override for this option as well?
No.
By the way, how do you bootstrap your cluster? Is it cephadm based?
This one is bootstrapped (on Quincy) with ceph-ansible. And when the "ceph config set" change didn't make a difference, I did also make a point of cycling all my mons and osds (which shouldn't be necessary, but I figured I'd try that, just in case).
And I also confirmed this same issue, in Quincy, after the cluster was adopted into cephadm management. At that point, the behaviour was still unchanged.
It was only after I upgraded the cluster to Reef, with cephadm/ceph orch, that the problem went away.
Cheers, Florian _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Florian Haas
-
Florian Haas