On May 20, 2024, at 12:21 PM, Matthew Vernon <mvernon@wikimedia.org> wrote:
Hi,
I'm probably Doing It Wrong here, but. My hosts are in racks, and I wanted ceph to use that information from the get-go, so I tried to achieve this during bootstrap.
This has left me with a single sad pg: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 33m, current state unknown, last acting []
.mgr pool perhaps.
ceph osd tree shows that CRUSH picked up my racks OK, eg. -3 45.11993 rack B4 -2 45.11993 host moss-be1001 1 hdd 3.75999 osd.1 up 1.00000 1.00000
Please send the entire first 10 lines or so of `ceph osd tree`
I passed this config to bootstrap with --config:
[global] osd_crush_chooseleaf_type = 3
Why did you set that? 3 is an unusual value. AIUI most of the time the only reason to change this option is if one is setting up a single-node sandbox - and perhaps localpools create a rule using it. I suspect this is at least part of your problem.
Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe"
Is it your intent to use spinners for payload data and SSD for metadata?
I could presumably fix this up by editing the crushmap (to put the racks into the default bucket), but what did I do wrong? Was this not a reasonable thing to want to do with cephadm?
I'm running ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)
Thanks,
Matthew _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io