some ceph general questions about the design
Hello together, we want to create a productive ceph storage system in our datacenter in may this year with openstack and ucs and i tested a lot in my cep test enviroment, and i have some general questions. whats receommended? 1. shoud i use a raid controller a create for example a raid 5 with all disks on each osd server? or should i passtrough all disks to ceph osd? 2. if i have a 2 pyhsicaly node osd cluster, did i need 3 physicall mons? 3. if i have a 3 physically node osd cluster, did i need 5 physicall mons? 3. where i should in install the mgr? on osd or mon 4. where i should in install the rgw? on osd or mon OR on 1 or 2 separate machines? in my testlab i created 3 VMs osds with mgr installed, and 5 VMs mons , and 1 VM as rgw -> is this correct? thx in advance hfreidhof
1. Do not use raid for osd disks... 1 ods per disk 2-3. I would have 3 or more osd nodes... more is better for when you have issues or need maintenance. We use vms for mon nodes with mgr on each mon node. 5 is the recommended for a production cluster but you can be ok with 3 for a small cluster 4. Again we use vms for rgw and scale these to traffic needs. Sent from my iPhone
On Apr 20, 2020, at 1:08 PM, harald.freidhof@gmail.com wrote:
Hello together,
we want to create a productive ceph storage system in our datacenter in may this year with openstack and ucs and i tested a lot in my cep test enviroment, and i have some general questions.
whats receommended?
1. shoud i use a raid controller a create for example a raid 5 with all disks on each osd server? or should i passtrough all disks to ceph osd? 2. if i have a 2 pyhsicaly node osd cluster, did i need 3 physicall mons? 3. if i have a 3 physically node osd cluster, did i need 5 physicall mons? 3. where i should in install the mgr? on osd or mon 4. where i should in install the rgw? on osd or mon OR on 1 or 2 separate machines?
in my testlab i created 3 VMs osds with mgr installed, and 5 VMs mons , and 1 VM as rgw -> is this correct?
thx in advance hfreidhof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
harald.freidhof@gmail.com (harald.freidhof) writes:
1. shoud i use a raid controller a create for example a raid 5 with all disks on each osd server? or should i passtrough all disks to ceph osd?
Set up the raid controller in JBOD mode, and passthrough the disks.
2. if i have a 2 pyhsicaly node osd cluster, did i need 3 physicall mons?
You can colocate mons on OSDs as long as you have enough RAM - you may need to run a third mon somewhere else to have a quorum. Also, 2 node cluster isn't recommended at all. 3 for testing is fine or a backup/replication site, better 5 or more.
3. if i have a 3 physically node osd cluster, did i need 5 physicall mons?
No, you'll be fine with 1 or 3 mons (although 1 would be risky)
3. where i should in install the mgr? on osd or mon
You could put the mgr on a osd, it is used mostly for statistics and some housekeeping tasks, but not "critical".
4. where i should in install the rgw? on osd or mon OR on 1 or 2 separate machines?
I don't run RGW so I won't comment.
ok thx for the answers we will connect later nearly 20 kvm hosts on the ceph cluster - then i build a 3 node osd cluster and passtrough all disks and i install the mgr daemon on them - i build 3 seperate mon server and install here the rgw? right? a few question to the options: - osd crush chooseleaf type - can i set here 3? if the hosts are in different racks? is that recomended? - osd pool default size - can i set here 2 ? for ex if i need to maintainance a osd and need to shutdown? thx again in advance for you answers hfreidhof
Hi Harald, - then i build a 3 node osd cluster and passtrough all disks and i install the mgr daemon on them - i build 3 seperate mon server and install here the rgw? right? As other friends suggested you can use VMs for mgr, mon and rgw, they are not so IOPS intensive and they are very flexible as well, you can easily replace/displace/move/add them in the future. The fact that if you can install them on the same machine depends on your load and level of complexity, the more nodes you have the more complex infra you will have to manage. But separating them would make it more secure and easier to troubleshoot in case of disasters (God forbid!). - osd crush chooseleaf type - can i set here 3? if the hosts are in different racks? is that recomended? I don't know honestly, but I think changing choose and chooseleaf have some perquisites and specific consideration in cluster. Better to read and ask more about it. - osd pool default size - can i set here 2 ? for ex if i need to maintainance a osd and need to shutdown AFAIK default pool size below 3 is not recommended, if you want to do maintenance on some OSDs you can mark them as out and do your job. Thanks, Khodayar On Tue, Apr 21, 2020 at 12:14 AM <harald.freidhof@gmail.com> wrote:
ok thx for the answers
we will connect later nearly 20 kvm hosts on the ceph cluster
- then i build a 3 node osd cluster and passtrough all disks and i install the mgr daemon on them - i build 3 seperate mon server and install here the rgw? right?
a few question to the options: - osd crush chooseleaf type - can i set here 3? if the hosts are in different racks? is that recomended? - osd pool default size - can i set here 2 ? for ex if i need to maintainance a osd and need to shutdown?
thx again in advance for you answers hfreidhof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Harald, 1. shoud i use a raid controller a create for example a raid 5 with all disks on each osd server? or should i passtrough all disks to ceph osd? If your OSD servers only contain flash drives, I'd suggest you go with a cheap HBA and configure it in JBOD mode (pass-though) If your OSD servers have HDDs, buy a good RAID Controller with a battery-backed write cache and configure it using multiple RAID-0 volumes (1 physical disk per volume). That way, reads and write will be accelerated by the cache on the HBA. 2. if i have a 2 pyhsicaly node osd cluster, did i need 3 physicall mons? Go with a minimum of 3 OSD servers (you'll end-up with 2 in case one dies) and 3 MONs 3. if i have a 3 physically node osd cluster, did i need 5 physicall mons? No. 3 MON are enough 3. where i should in install the mgr? on osd or mon MON 4. where i should in install the rgw? on osd or mon OR on 1 or 2 separate machines? Here is my suggestion: buy 3 standalone KVM servers and install your MONs, RGW and whatever load-balancing servers (i.e. HAProxy) as guests on these KVM servers. Make sure the guests are properly balanced against you 3 KVM servers so in case one dies, it does not affect the service. ________________________________ From: harald.freidhof@gmail.com <harald.freidhof@gmail.com> Sent: Monday, April 20, 2020 3:07 PM To: ceph-users@ceph.io <ceph-users@ceph.io> Subject: [ceph-users] some ceph general questions about the design Hello together, we want to create a productive ceph storage system in our datacenter in may this year with openstack and ucs and i tested a lot in my cep test enviroment, and i have some general questions. whats receommended? 1. shoud i use a raid controller a create for example a raid 5 with all disks on each osd server? or should i passtrough all disks to ceph osd? 2. if i have a 2 pyhsicaly node osd cluster, did i need 3 physicall mons? 3. if i have a 3 physically node osd cluster, did i need 5 physicall mons? 3. where i should in install the mgr? on osd or mon 4. where i should in install the rgw? on osd or mon OR on 1 or 2 separate machines? in my testlab i created 3 VMs osds with mgr installed, and 5 VMs mons , and 1 VM as rgw -> is this correct? thx in advance hfreidhof _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Antoine Lecrux
-
Freddy Andersen
-
harald.freidhof＠gmail.com
-
Khodayar Doustar
-
Phil Regnauld