See I have total host 5, each host holding 24 HDD and each HDD is of size 9.1TiB. So, a total of 1.2PiB out of which i am getting 700TiB. I did erasure coding 3+2 and placement group 128. But, the issue i am facing is when I turn off one node write is completely disabled. Erasure coding 3+2 can handle two nodes failure but it's not working in my case. I request this community to help me tackle this issue. The min size is 3 and 4 pools are there.
Hi, We add hosts to the cluster using fqdn, manually (ceph orch host add) everything works fine. However, if we use the spec file as below, the whole thing falls apart. Ceph 18.2.4 --- service_type: host addr: xx.xx.xx.xx hostname: ceph001.xx002.xx.xx.xx.com location: root: xx002 rack: rack01 labels: - osd - rgw --- service_type: osd service_id: object_hdd service_name: osd.object_hdd placement: host_pattern: ceph* crush_device_class: object_hdd spec: data_devices: rotational: 1 db_devices: rotational: 0 size: '3000G:' --- service_type: osd service_id: index_nvme service_name: osd.index_nvme placement: host_pattern: ceph* crush_device_class: index_nvme spec: data_devices: rotational: 0 size: ':900G' Applying this spec results in two hosts, one fqdn and the other short: root@mon001(xx002):~/cephadm# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -4 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root dias002 -3 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - rack rack01 -2 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph001.xx002.xx.xx.xx.com -1 662.71497 - 663 TiB 7.0 TiB 102 MiB 37 KiB 1.7 GiB 656 TiB 1.05 1.00 - root default -9 662.71497 - 663 TiB 7.0 TiB 102 MiB 37 KiB 1.7 GiB 656 TiB 1.05 1.00 - host ceph001 36 index_nvme 0.87329 1.00000 894 GiB 33 MiB 2.7 MiB 1 KiB 30 MiB 894 GiB 0.00 0.00 0 up osd.36 0 object_hdd 18.38449 1.00000 18 TiB 199 GiB 2.7 MiB 1 KiB 56 MiB 18 TiB 1.06 1.00 0 up osd.0 1 object_hdd 18.38449 1.00000 18 TiB 199 GiB 2.7 MiB 1 KiB 74 MiB 18 TiB 1.06 1.00 0 up osd.1 2 object_hdd 18.38449 1.00000 18 TiB 199 GiB 2.7 MiB 1 KiB 56 MiB 18 TiB 1.06 1.00 0 up osd.2 This looks like a bug, but I'm not sure, maybe someone has encountered something similar? Regards, Piotr
First, min_size=3 with k=3 is not a good idea. You don’t provide any details, so it’s difficult to give any reasonable explanation. I’ll go with a wrong crush rule that doesn’t have host as failure domain but OSD. Zitat von Deba Dey <debadey886@gmail.com>:
See I have total host 5, each host holding 24 HDD and each HDD is of size 9.1TiB. So, a total of 1.2PiB out of which i am getting 700TiB. I did erasure coding 3+2 and placement group 128. But, the issue i am facing is when I turn off one node write is completely disabled. Erasure coding 3+2 can handle two nodes failure but it's not working in my case. I request this community to help me tackle this issue. The min size is 3 and 4 pools are there. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Deba Dey
-
Eugen Block
-
Piotr Pisz