[ceph-users] cephadm bootstraps cluster with bad CRUSH map(?)

20 May 2024

      Hi,

I'm probably Doing It Wrong here, but. My hosts are in racks, and I 
wanted ceph to use that information from the get-go, so I tried to 
achieve this during bootstrap.

This has left me with a single sad pg:
[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive
     pg 1.0 is stuck inactive for 33m, current state unknown, last acting []

ceph osd tree shows that CRUSH picked up my racks OK, eg.
-3          45.11993  rack B4
-2          45.11993      host moss-be1001
  1    hdd    3.75999          osd.1             up   1.00000  1.00000

But root seems empty:
-1                 0  root default

and if I decompile the crush map, indeed:
# buckets
root default {
         id -1           # do not change unnecessarily
         id -14 class hdd                # do not change unnecessarily
         # weight 0.00000
         alg straw2
         hash 0  # rjenkins1
}

which does indeed look empty, whereas I have rack entries that contain 
the relevant hosts.

And the replication rule:
rule replicated_rule {
         id 0
         type replicated
         step take default
         step chooseleaf firstn 0 type rack
         step emit
}

I passed this config to bootstrap with --config:

[global]
   osd_crush_chooseleaf_type = 3

and an initial spec file with host entries like this:

service_type: host
hostname: moss-be1001
addr: 10.64.16.40
location:
   rack: B4
labels:
   - _admin
   - NVMe

Once the cluster was up I used an osd spec file that looked like:
service_type: osd
service_id: rrd_single_NVMe
placement:
   label: "NVMe"
spec:
   data_devices:
     rotational: 1
   db_devices:
     model: "NVMe"

I could presumably fix this up by editing the crushmap (to put the racks 
into the default bucket), but what did I do wrong? Was this not a 
reasonable thing to want to do with cephadm?

I'm running
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)

Thanks,

Matthew

[ceph-users] cephadm bootstraps cluster with bad CRUSH map(?)

Matthew Vernon