Hi, I found my error, it was a mismatch between the monitor network ip address and the --cluster_network which were in different subnets. I misunderstood the --cluster_network subnet, I thought that when creating a cluster, the monitor IP designed the public Network, and if I wanted to separate public and private (cluster) networks, I needed to use the --cluster_network option. Maybe I was in over my head, but sometimes it is not that clear. Regards. Le ven. 15 mars 2024 à 07:18, wodel youchi <wodel.youchi@gmail.com> a écrit :

Hi,

Note : Firewall is disabled on all hosts.

Regards.

Le ven. 15 mars 2024 à 06:42, wodel youchi <wodel.youchi@gmail.com> a écrit :

...

Hi,

I did recreate the cluster again, and it is the result.

This is my initial bootstrap

cephadm --image 192.168.2.36:4000/ceph/ceph:v18 bootstrap --initial-dashboard-user admin \ --initial-dashboard-password adminpass --dashboard-password-noupdate --registry-url 192.168.2.36:4000 \ --registry-username admin --registry-password admin --mon-ip 20.1.0.23 --cluster-network 20.2.0.0/16 \ --ssh-private-key /root/.ssh/id_rsa --ssh-public-key /root/.ssh/id_rsa.pub \ -c initial-ceph.conf

This is my initial-ceph.conf [mgr] mgr/cephadm/container_image_prometheus = 192.168.2.36:4000/prometheus/prometheus:v2.43.0 mgr/cephadm/container_image_node_exporter = 192.168.2.36:4000/prometheus/node-exporter:v1.5.0 mgr/cephadm/container_image_grafana = 192.168.2.36:4000/ceph/ceph-grafana:9.4.7 mgr/cephadm/container_image_alertmanager = 192.168.2.36:4000/prometheus/alertmanager:v0.25.0

Then I added two managers and monitors # ceph orch host add controllerb 20.1.0.27 _admin # ceph orch host add controllerc 20.1.0.31 _admin # ceph orch apply mon --placement="3 controllera controllerb controllerc" # ceph orch apply mgr --placement="3 controllera controllerb controllerc"

Then I added node-exporter, prometheus, grafana and crash Then I added osd hosts # ceph orch host add computehci01 20.1.0.2 # ceph orch host add computehci02 20.1.0.3 # ceph orch host add computehci03 20.1.0.4 ... # ceph orch host add computehci09 20.1.0.10 ...

And finally I added osd daemons # ceph orch daemon add osd computehci01:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1 # ceph orch daemon add osd computehci02:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1 ...

I created a pool # ceph osd pool create volumes replicated # ceph osd pool application enable volumes rbd

I even created cephfs pools and added mds service, but still 100% pgs are unknown???? !!!!

[root@controllera ~]# ceph -s cluster: id: df914aa2-e21a-11ee-b8df-3cecef2872f0 health: HEALTH_WARN 1 MDSs report slow metadata IOs Reduced data availability: 4 pgs inactive

services: mon: 3 daemons, quorum controllera,controllerc,controllerb (age 13h) mgr: controllera.ajttxz(active, since 13h), standbys: controllerb.qtixeq, controllerc.pqyqqo mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 7h), 36 in (since 7h)

data: volumes: 1/1 healthy pools: 4 pools, 4 pgs objects: 0 objects, 0 B usage: 1.1 GiB used, 110 TiB / 110 TiB avail pgs: 100.000% pgs unknown 4 unknown

[root@controllera ~]# ceph osd pool ls detail pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 46 flags hashpspool,creating stripe_width 0 pg_num_max 32 pg_nu m_min 1 application mgr pool 2 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 154 flags hashpspool,creating stripe_width 0 application rbd pool 3 'cephfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157 flags hashpspool,creating stripe_width 0 application ceph fs pool 4 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157 flags hashpspool,creating stripe_width 0 pg_auto scale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs

What am I missing, why PGs won't pair?

Regards.

Le jeu. 14 mars 2024 à 15:36, wodel youchi <wodel.youchi@gmail.com> a écrit :

...

Hi,

I am creating a new ceph cluster using REEF.

This is my host_specs file [root@controllera config]# cat hosts-specs2.yml service_type: host hostname: computehci01 addr: 20.1.0.2 location: chassis: chassis1 --- service_type: host hostname: computehci02 addr: 20.1.0.3 location: chassis: chassis1 --- service_type: host hostname: computehci03 addr: 20.1.0.4 location: chassis: chassis1 --- service_type: host hostname: computehci04 addr: 20.1.0.5 location: chassis: chassis2 --- service_type: host hostname: computehci05 addr: 20.1.0.6 location: chassis: chassis2 --- service_type: host hostname: computehci06 addr: 20.1.0.7 location: chassis: chassis2 --- service_type: host hostname: computehci07 addr: 20.1.0.8 location: chassis: chassis3 --- service_type: host hostname: computehci08 addr: 20.1.0.9 location: chassis: chassis3 --- service_type: host hostname: computehci09 addr: 20.1.0.10 location: chassis: chassis3 --- service_type: host hostname: computehci10 addr: 20.1.0.11 location: chassis: chassis3 --- service_type: host hostname: computehci11 addr: 20.1.0.12 location: chassis: chassis4 --- service_type: host hostname: computehci12 addr: 20.1.0.13 location: chassis: chassis4 --- service_type: host hostname: computehci13 addr: 20.1.0.14 location: chassis: chassis4 --- service_type: host hostname: computehci14 addr: 20.1.0.15 location: chassis: chassis4 --- service_type: host hostname: computehci15 addr: 20.1.0.16 location: chassis: chassis5 --- service_type: host hostname: computehci16 addr: 20.1.0.17 location: chassis: chassis5 --- service_type: host hostname: computehci17 addr: 20.1.0.18 location: chassis: chassis5 --- service_type: host hostname: computehci18 addr: 20.1.0.19 location: chassis: chassis5 --- service_type: host hostname: computehci19 addr: 20.1.0.20 location: chassis: chassis6 --- service_type: host hostname: computehci20 addr: 20.1.0.21 location: chassis: chassis6 --- service_type: host hostname: computehci21 addr: 20.1.0.22 location: chassis: chassis6 --- service_type: host hostname: computehci22 addr: 20.1.0.24 location: chassis: chassis7 --- service_type: host hostname: computehci23 addr: 20.1.0.25 location: chassis: chassis7 --- service_type: host hostname: computehci24 addr: 20.1.0.26 location: chassis: chassis7 --- service_type: host hostname: computehci25 addr: 20.1.0.28 location: chassis: chassis8 --- service_type: host hostname: computehci26 addr: 20.1.0.29 location: chassis: chassis8 --- service_type: host hostname: computehci27 addr: 20.1.0.30 location: chassis: chassis8 --- service_type: host hostname: controllera addr: 20.1.0.23 --- service_type: host hostname: controllerb addr: 20.1.0.27 --- service_type: host hostname: controllerc addr: 20.1.0.31 --- service_type: mon placement: hosts: - controllera - controllerb - controllerc --- service_type: mgr placement: hosts: - controllera - controllerb - controllerc --- service_type: osd service_id: default_drive_group placement: hosts: - computehci01 - computehci02 - computehci03 - computehci04 - computehci05 - computehci06 - computehci07 - computehci08 - computehci09 - computehci10 - computehci11 - computehci12 - computehci13 - computehci14 - computehci15 - computehci16 - computehci17 - computehci18 - computehci19 - computehci20 - computehci21 - computehci22 - computehci23 - computehci24 - computehci25 - computehci26 - computehci27 spec: data_devices: rotational: 0

All osds were added but, pg still unknown state

I've created a pool, but it didn't change anything.

[root@controllerb ~]# ceph -s cluster: id: be250ade-e1f2-11ee-a6ff-3cecef2872f0 health: HEALTH_WARN Reduced data availability: 1 pg inactive

services: mon: 3 daemons, quorum controllera,controllerc,controllerb (age 3h) mgr: controllerc.jevbkl(active, since 21s), standbys: controllera.zwlolp, controllerb.vqkdga osd: 108 osds: 108 up (since 2m), 108 in (since 24m)

data: pools: 2 pools, 33 pgs objects: 0 objects, 0 B usage: 5.1 GiB used, 330 TiB / 330 TiB avail

* pgs: 100.000% pgs unknown 33 unknown*

Did I miss something?

Regards.

Hi, Thanks Stefan. Yes I do have separate port-channel interfaces, for public and cluster networks. I just didn't understand the documentation (which is sometimes not that clear). For me when you put forward the --cluster_network option, it meant that --mon-ip was in the public network by default and that you want to define a separate network for your cluster to use as private net. I was reading the documentation when I came across : *ceph config set mon public_network* command, it's then when I saw what could be my mistake. I redid the deployment and so far so good :-) I did use *ceph config set mon public_network* to specify my public net, now I have to test my clients if they can connect to it. Regards. Le ven. 15 mars 2024 à 08:20, Stefan Kooman <stefan@bit.nl> a écrit :

On 15-03-2024 08:10, wodel youchi wrote:

...

Hi,

I found my error, it was a mismatch between the monitor network ip address and the --cluster_network which were in different subnets. I misunderstood the --cluster_network subnet, I thought that when creating a cluster, the monitor IP designed the public Network, and if I wanted to separate public and private (cluster) networks, I needed to use the --cluster_network option. Maybe I was in over my head, but sometimes it is not that clear.

Ah, good that you found the error. Do you have separate interfaces / port-channels for public and cluster networks? Note that a cluster network is not required and makes things more complicated (as you have noted) and might not give you any benefits when the infrastructure does not match this separation of functions.

Gr. Stefan