Hi everyone, I like to know how many pool should I create for multiple cephfs ? Knowing I got two class of osd (hdd and ssd), and I have a need of ~ 20/30 cephfs (currently and that number will increase with time). Should I create one cephfs_metadata_replicated one cephfs_data_replicated few cephfs_data_erasure_coding (depending of k/m) and put all my cephfs inside two of them. Or should I create for each cephfs a couple of pool metadata/data ? Il will also need to have ceph S3 storage, same question, should I have a designated pool for S3 storage or can/should I use the same cephfs_data_replicated/erasure pool ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 24 janv. 2024 09:33:09 CET
Hi, On 1/24/24 09:40, Albert Shih wrote:
Knowing I got two class of osd (hdd and ssd), and I have a need of ~ 20/30 cephfs (currently and that number will increase with time).
Why do you need 20 - 30 separate CephFS instances?
and put all my cephfs inside two of them. Or should I create for each cephfs a couple of pool metadata/data ?
Each CephFS instance needs their own pools, at least two (data + metadata) per instance. And each CephFS needs at least one MDS running, better with an additional cold or even hot standby MDS.
Il will also need to have ceph S3 storage, same question, should I have a designated pool for S3 storage or can/should I use the same cephfs_data_replicated/erasure pool ?
No, S3 needs its own pools. It cannot re-use CephFS pools. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Le 24/01/2024 à 09:45:56+0100, Robert Sander a écrit Hi
On 1/24/24 09:40, Albert Shih wrote:
Knowing I got two class of osd (hdd and ssd), and I have a need of ~ 20/30 cephfs (currently and that number will increase with time).
Why do you need 20 - 30 separate CephFS instances?
99.99% because I'm newbie with ceph and don't understand clearly how the autorisation work with cephfs ;-) If I say 20-30 it's because I currently have on my classic ZFS/NFS server around 25 «datasets» exported to various server. But because you question I understand I can put many export «inside» one cephfs.
and put all my cephfs inside two of them. Or should I create for each cephfs a couple of pool metadata/data ?
Each CephFS instance needs their own pools, at least two (data + metadata) per instance. And each CephFS needs at least one MDS running, better with an additional cold or even hot standby MDS.
Ok. I got for my ceph cluster two set of servers, first set are for services (mgr,mon,etc.) with ssd and don't currently run any osd (but still have 2 ssd not used), I also got a second set of server with HDD and 2 SSD. The data pool will be on the second set (with HDD). Where should I run the MDS and on which osd ?
Il will also need to have ceph S3 storage, same question, should I have a designated pool for S3 storage or can/should I use the same cephfs_data_replicated/erasure pool ?
No, S3 needs its own pools. It cannot re-use CephFS pools.
Ok thanks. Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 24 janv. 2024 09:55:26 CET
Hi Albert, In this scenario, it is more consistent to work with subvolumes. Regarding security, you can use namespaces to isolate access at the OSD level. What Robert emphasizes is that creating pools dynamically is not without effect on the number of PGs and (therefore) on the architecture (PG per OSD, balancer, pg autoscaling, etc.) ________________________________________________________ Cordialement, *David CASIER* ________________________________________________________ Le mer. 24 janv. 2024 à 10:10, Albert Shih <Albert.Shih@obspm.fr> a écrit :
Le 24/01/2024 à 09:45:56+0100, Robert Sander a écrit Hi
On 1/24/24 09:40, Albert Shih wrote:
Knowing I got two class of osd (hdd and ssd), and I have a need of ~
20/30
cephfs (currently and that number will increase with time).
Why do you need 20 - 30 separate CephFS instances?
99.99% because I'm newbie with ceph and don't understand clearly how the autorisation work with cephfs ;-)
If I say 20-30 it's because I currently have on my classic ZFS/NFS server around 25 «datasets» exported to various server.
But because you question I understand I can put many export «inside» one cephfs.
and put all my cephfs inside two of them. Or should I create for each cephfs a couple of pool metadata/data ?
Each CephFS instance needs their own pools, at least two (data + metadata) per instance. And each CephFS needs at least one MDS running, better with an additional cold or even hot standby MDS.
Ok. I got for my ceph cluster two set of servers, first set are for services (mgr,mon,etc.) with ssd and don't currently run any osd (but still have 2 ssd not used), I also got a second set of server with HDD and 2 SSD. The data pool will be on the second set (with HDD). Where should I run the MDS and on which osd ?
Il will also need to have ceph S3 storage, same question, should I
have a
designated pool for S3 storage or can/should I use the same cephfs_data_replicated/erasure pool ?
No, S3 needs its own pools. It cannot re-use CephFS pools.
Ok thanks.
Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 24 janv. 2024 09:55:26 CET _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 24/01/2024 à 10:23:20+0100, David C. a écrit Hi,
In this scenario, it is more consistent to work with subvolumes.
Ok. I will do that.
Regarding security, you can use namespaces to isolate access at the OSD level.
Hum....I'm currently have no idea what you just say.... but that's OK ;-)
What Robert emphasizes is that creating pools dynamically is not without effect on the number of PGs and (therefore) on the architecture (PG per OSD, balancer, pg autoscaling, etc.)
Ok.....no worries....I didn't know it was possible....;-) Regards. JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 24 janv. 2024 10:31:44 CET
Hi, On 1/24/24 10:08, Albert Shih wrote:
99.99% because I'm newbie with ceph and don't understand clearly how the autorisation work with cephfs ;-)
I strongly recommend you to ask for a expierenced Ceph consultant that helps you design and setup your storage cluster. It looks like you try to make design decisions that will heavily influence performance of the system.
If I say 20-30 it's because I currently have on my classic ZFS/NFS server around 25 «datasets» exported to various server.
The next question is how would the "consumers" access the filesystem: Via NFS or mounted directly. Even with the second option you can separate client access via CephX keys as David already wrote.
Ok. I got for my ceph cluster two set of servers, first set are for services (mgr,mon,etc.) with ssd and don't currently run any osd (but still have 2 ssd not used), I also got a second set of server with HDD and 2 SSD. The data pool will be on the second set (with HDD). Where should I run the MDS and on which osd ?
Do you intend to use the Ceph cluster only for archival storage? Hwo large is your second set of Ceph nodes, how many HDDs in each? Do you intend to use the SSDs for the OSDs' RocksDB? Where do you plan to store the metadata pools for CephFS? They should be stored on fats media. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Le 24/01/2024 à 10:33:45+0100, Robert Sander a écrit Hi,
On 1/24/24 10:08, Albert Shih wrote:
99.99% because I'm newbie with ceph and don't understand clearly how the autorisation work with cephfs ;-)
I strongly recommend you to ask for a expierenced Ceph consultant that helps you design and setup your storage cluster.
I known I'm working on (meaning I'm waiting my administration to do «what need to be done)...
It looks like you try to make design decisions that will heavily influence performance of the system.
I'm well aware....
If I say 20-30 it's because I currently have on my classic ZFS/NFS server around 25 «datasets» exported to various server.
The next question is how would the "consumers" access the filesystem: Via NFS or mounted directly. Even with the second option you can separate client access via CephX keys as David already wrote.
The separate client key would be more than enough for us.
Ok. I got for my ceph cluster two set of servers, first set are for services (mgr,mon,etc.) with ssd and don't currently run any osd (but still have 2 ssd not used), I also got a second set of server with HDD and 2 SSD. The data pool will be on the second set (with HDD). Where should I run the MDS and on which osd ?
Do you intend to use the Ceph cluster only for archival storage?
Mostly yes.
Hwo large is your second set of Ceph nodes, how many HDDs in each? Do you
Huge ;-) I got 6 ceph server with ... 60 HDD. (I know, I know it's not ideal)
intend to use the SSDs for the OSDs' RocksDB?
RocksDB ? no...
Where do you plan to store the metadata pools for CephFS? They should be
That's exactly the question... My cluster are : 5 server with «small» ssd for service (each got 2 ssd no currently used) 6 server with «huge» HDD for data (each got 2 ssd no currently used) so for my cephfs metadata I can put them on my 5 servers for services (but that's mean the mds running on those 5 servers) or should I use the ssd on the 6 server who hold the OSD for data Thanks. Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 24 janv. 2024 10:48:11 CET
participants (3)
-
Albert Shih
-
David C.
-
Robert Sander