cephadm bootstraps cluster with bad CRUSH map(?)
Hi, I'm probably Doing It Wrong here, but. My hosts are in racks, and I wanted ceph to use that information from the get-go, so I tried to achieve this during bootstrap. This has left me with a single sad pg: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 33m, current state unknown, last acting [] ceph osd tree shows that CRUSH picked up my racks OK, eg. -3 45.11993 rack B4 -2 45.11993 host moss-be1001 1 hdd 3.75999 osd.1 up 1.00000 1.00000 But root seems empty: -1 0 root default and if I decompile the crush map, indeed: # buckets root default { id -1 # do not change unnecessarily id -14 class hdd # do not change unnecessarily # weight 0.00000 alg straw2 hash 0 # rjenkins1 } which does indeed look empty, whereas I have rack entries that contain the relevant hosts. And the replication rule: rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type rack step emit } I passed this config to bootstrap with --config: [global] osd_crush_chooseleaf_type = 3 and an initial spec file with host entries like this: service_type: host hostname: moss-be1001 addr: 10.64.16.40 location: rack: B4 labels: - _admin - NVMe Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" I could presumably fix this up by editing the crushmap (to put the racks into the default bucket), but what did I do wrong? Was this not a reasonable thing to want to do with cephadm? I'm running ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Thanks, Matthew
On May 20, 2024, at 12:21 PM, Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi,
I'm probably Doing It Wrong here, but. My hosts are in racks, and I wanted ceph to use that information from the get-go, so I tried to achieve this during bootstrap.
This has left me with a single sad pg: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 33m, current state unknown, last acting []
.mgr pool perhaps.
ceph osd tree shows that CRUSH picked up my racks OK, eg. -3 45.11993 rack B4 -2 45.11993 host moss-be1001 1 hdd 3.75999 osd.1 up 1.00000 1.00000
Please send the entire first 10 lines or so of `ceph osd tree`
I passed this config to bootstrap with --config:
[global] osd_crush_chooseleaf_type = 3
Why did you set that? 3 is an unusual value. AIUI most of the time the only reason to change this option is if one is setting up a single-node sandbox - and perhaps localpools create a rule using it. I suspect this is at least part of your problem.
Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe"
Is it your intent to use spinners for payload data and SSD for metadata?
I could presumably fix this up by editing the crushmap (to put the racks into the default bucket), but what did I do wrong? Was this not a reasonable thing to want to do with cephadm?
I'm running ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)
Thanks,
Matthew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 20/05/2024 17:29, Anthony D'Atri wrote:
On May 20, 2024, at 12:21 PM, Matthew Vernon <mvernon@wikimedia.org> wrote:
This has left me with a single sad pg: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 33m, current state unknown, last acting []
.mgr pool perhaps.
I think so
ceph osd tree shows that CRUSH picked up my racks OK, eg. -3 45.11993 rack B4 -2 45.11993 host moss-be1001 1 hdd 3.75999 osd.1 up 1.00000 1.00000
Please send the entire first 10 lines or so of `ceph osd tree`
root@moss-be1001:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -7 176.11194 rack F3 -6 176.11194 host moss-be1003 2 hdd 7.33800 osd.2 up 1.00000 1.00000 3 hdd 7.33800 osd.3 up 1.00000 1.00000 6 hdd 7.33800 osd.6 up 1.00000 1.00000 9 hdd 7.33800 osd.9 up 1.00000 1.00000 12 hdd 7.33800 osd.12 up 1.00000 1.00000 13 hdd 7.33800 osd.13 up 1.00000 1.00000 16 hdd 7.33800 osd.16 up 1.00000 1.00000 19 hdd 7.33800 osd.19 up 1.00000 1.00000
I passed this config to bootstrap with --config:
[global] osd_crush_chooseleaf_type = 3
Why did you set that? 3 is an unusual value. AIUI most of the time the only reason to change this option is if one is setting up a single-node sandbox - and perhaps localpools create a rule using it. I suspect this is at least part of your problem.
I wanted to have rack as failure domain rather than host i.e. to ensure that each replica goes in a different rack (academic at the moment as I have 3 hosts, one in each rack, but for future expansion important).
Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe"
Is it your intent to use spinners for payload data and SSD for metadata?
Yes. Regards, Matthew
This has left me with a single sad pg: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 33m, current state unknown, last acting []
.mgr pool perhaps.
I think so
ceph osd tree shows that CRUSH picked up my racks OK, eg. -3 45.11993 rack B4 -2 45.11993 host moss-be1001 1 hdd 3.75999 osd.1 up 1.00000 1.00000 Please send the entire first 10 lines or so of `ceph osd tree`
root@moss-be1001:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -7 176.11194 rack F3 -6 176.11194 host moss-be1003 2 hdd 7.33800 osd.2 up 1.00000 1.00000 3 hdd 7.33800 osd.3 up 1.00000 1.00000 6 hdd 7.33800 osd.6 up 1.00000 1.00000 9 hdd 7.33800 osd.9 up 1.00000 1.00000 12 hdd 7.33800 osd.12 up 1.00000 1.00000 13 hdd 7.33800 osd.13 up 1.00000 1.00000 16 hdd 7.33800 osd.16 up 1.00000 1.00000 19 hdd 7.33800 osd.19 up 1.00000 1.00000
Yep. Your racks and thus hosts and OSDs aren’t under the `default` or any other root, so they won’t get picked by any CRUSH rule.
I passed this config to bootstrap with --config:
[global] osd_crush_chooseleaf_type = 3
Why did you set that? 3 is an unusual value. AIUI most of the time the only reason to change this option is if one is setting up a single-node sandbox - and perhaps localpools create a rule using it. I suspect this is at least part of your problem.
I wanted to have rack as failure domain rather than host i.e. to ensure that each replica goes in a different rack (academic at the moment as I have 3 hosts, one in each rack, but for future expansion important).
You do that with the CRUSH rule, not with osd_crush_chooseleaf_type. Set that back to the default value of `1`. This option is marked `dev` for a reason ;) And the replication rule: rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type rack ###### `rack` here is what selects the failure domain. step emit }
I could presumably fix this up by editing the crushmap (to put the racks into the default bucket)
That would probably help `ceph osh crush move F3 root=default` but I think you’d also need to revert `osd_crush_chooseleaf_type` too. Might be better to wipe and redeploy so you know that down the road when you add / replace hardware this behavior doesn’t resurface.
Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" Is it your intent to use spinners for payload data and SSD for metadata?
Yes.
You might want to set `db_slots` accordingly, by default I think it’ll be 1:1 which probably isn’t what you intend.
Regards,
Matthew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Thanks for your help! On 20/05/2024 18:13, Anthony D'Atri wrote:
You do that with the CRUSH rule, not with osd_crush_chooseleaf_type. Set that back to the default value of `1`. This option is marked `dev` for a reason ;)
OK [though not obviously at https://docs.ceph.com/en/reef/rados/configuration/pool-pg-config-ref/#confva... ]
but I think you’d also need to revert `osd_crush_chooseleaf_type` too. Might be better to wipe and redeploy so you know that down the road when you add / replace hardware this behavior doesn’t resurface.
Yep, I'm still at the destroy-and-recreate point here, trying to make sure I can do this repeatably.
Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" Is it your intent to use spinners for payload data and SSD for metadata?
Yes.
You might want to set `db_slots` accordingly, by default I think it’ll be 1:1 which probably isn’t what you intend.
Is there an easy way to check this? The docs suggested it would work, and vgdisplay on the vg that pvs tells me the nvme device is in shows 24 LVs... Thanks, Matthew
On May 20, 2024, at 2:24 PM, Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi,
Thanks for your help!
On 20/05/2024 18:13, Anthony D'Atri wrote:
You do that with the CRUSH rule, not with osd_crush_chooseleaf_type. Set that back to the default value of `1`. This option is marked `dev` for a reason ;)
OK [though not obviously at https://docs.ceph.com/en/reef/rados/configuration/pool-pg-config-ref/#confva... ]
Aye that description is pretty oblique. I’d update it if I fully understood it. But one might argue that if you don’t understand something, leave it alone ;)
but I think you’d also need to revert `osd_crush_chooseleaf_type` too. Might be better to wipe and redeploy so you know that down the road when you add / replace hardware this behavior doesn’t resurface.
Yep, I'm still at the destroy-and-recreate point here, trying to make sure I can do this repeatably.
Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" Is it your intent to use spinners for payload data and SSD for metadata?
Yes. You might want to set `db_slots` accordingly, by default I think it’ll be 1:1 which probably isn’t what you intend.
Is there an easy way to check this? The docs suggested it would work, and vgdisplay on the vg that pvs tells me the nvme device is in shows 24 LVs...
If you create the OSDs and their DB/WAL devices show NVMe partitions then you’re good. How many NVMe devices do you have on the HDD nodes?
Thanks,
Matthew
Hi, Returning to this, it looks like the issue wasn't to do with how osd_crush_chooseleaf_type ; I destroyed and re-created my cluster as before, and I have the same problem again: pg 1.0 is stuck inactive for 10m, current state unknown, last acting [] as before, ceph osd tree: root@moss-be1001:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -7 176.11194 rack F3 -6 176.11194 host moss-be1003 13 hdd 7.33800 osd.13 up 1.00000 1.00000 15 hdd 7.33800 osd.15 up 1.00000 1.00000 And checking the crushmap, the default bucket is again empty: root default { id -1 # do not change unnecessarily id -14 class hdd # do not change unnecessarily # weight 0.00000 alg straw2 hash 0 # rjenkins1 } [by way of confirming that I didn't accidentally leave the old config fragment lying around, the replication rule has: step chooseleaf firstn 0 type host ] So it looks like setting location: in my spec is breaking the cluster bootstrap - the hosts aren't put into default, but neither are the declared racks. As a reminder, that spec has host entries like: service_type: host hostname: moss-be1003 addr: 10.64.136.22 location: rack: F3 labels: - _admin - NVMe Is this expected behaviour? Presumably I can fix the cluster by using "ceph osd crush move F3 root=default" and similar for the others, but is there a way to have what I want done by cephadm bootstrap? Thanks, Matthew
Hi, you can specify the entire tree in the location statement, if you need to: ceph:~ # cat host-spec.yaml service_type: host hostname: ceph addr: <IP> location: root: default rack: rack2 and after the bootstrap it looks like expected: ceph:~ # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0 root default -3 0 rack rack2 -2 0 host ceph This should be made a bit clearer in the docs [0], I added Zac. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephadm/host-management/#setting-the-initial... Zitat von Matthew Vernon <mvernon@wikimedia.org>:
Hi,
Returning to this, it looks like the issue wasn't to do with how osd_crush_chooseleaf_type ; I destroyed and re-created my cluster as before, and I have the same problem again:
pg 1.0 is stuck inactive for 10m, current state unknown, last acting []
as before, ceph osd tree:
root@moss-be1001:/# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -7 176.11194 rack F3 -6 176.11194 host moss-be1003 13 hdd 7.33800 osd.13 up 1.00000 1.00000 15 hdd 7.33800 osd.15 up 1.00000 1.00000
And checking the crushmap, the default bucket is again empty:
root default { id -1 # do not change unnecessarily id -14 class hdd # do not change unnecessarily # weight 0.00000 alg straw2 hash 0 # rjenkins1 }
[by way of confirming that I didn't accidentally leave the old config fragment lying around, the replication rule has: step chooseleaf firstn 0 type host ]
So it looks like setting location: in my spec is breaking the cluster bootstrap - the hosts aren't put into default, but neither are the declared racks. As a reminder, that spec has host entries like:
service_type: host hostname: moss-be1003 addr: 10.64.136.22 location: rack: F3 labels: - _admin - NVMe
Is this expected behaviour? Presumably I can fix the cluster by using "ceph osd crush move F3 root=default" and similar for the others, but is there a way to have what I want done by cephadm bootstrap?
Thanks,
Matthew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 22/05/2024 12:44, Eugen Block wrote:
you can specify the entire tree in the location statement, if you need to:
[snip] Brilliant, that's just the ticket, thank you :)
This should be made a bit clearer in the docs [0], I added Zac.
I've opened a MR to update the docs, I hope it's at least useful as a starter-for-ten: https://github.com/ceph/ceph/pull/57633 Thanks, Matthew
Hi, thanks for picking that up so quickly! I haven't used a host spec file yet to add new hosts, but if you read my thread about the unknown PGs, this might be my first choice to do that in the future. So thanks again for bringig it to my attention. ;-) Regards, Eugen Zitat von Matthew Vernon <mvernon@wikimedia.org>:
Hi,
On 22/05/2024 12:44, Eugen Block wrote:
you can specify the entire tree in the location statement, if you need to:
[snip]
Brilliant, that's just the ticket, thank you :)
This should be made a bit clearer in the docs [0], I added Zac.
I've opened a MR to update the docs, I hope it's at least useful as a starter-for-ten: https://github.com/ceph/ceph/pull/57633
Thanks,
Matthew
participants (3)
-
Anthony D'Atri
-
Eugen Block
-
Matthew Vernon