Thanks. For replica, what is the best way to change crush profile ? Is it to create a new replica profile, and set this profile as crush rulest for the pool (something like ceph osd pool set {pool-name} crush_ruleset my_new_rule) ? For erasure coding, I would thus have to change the profile at least to k=3, m=3 (for now I only have 6 osd servers). But if I am correct, this cannot be changed for an existing pool and I will have to create a new pool and migrate all data from the current one to the new one. Is that correct ? F. Le 28/11/2019 à 17:51, Paul Emmerich a écrit :

Use a crush rule likes this for replica:

1) root default class XXX 2) choose 2 rooms 3) choose 2 disks

That'll get you 4 OSDs in two rooms and the first 3 of these get data, the fourth will be ignored. That guarantees that losing a room will lose you at most 2 out of 3 copies. This is for disaster recovery only, it'll guarantee durability if you lose a room but not availability.

3+2 erasure coding cannot be split across two rooms in this way because, well, you need 3 out of 5 shards to survive, so you cannot lose half of them.

Paul

Thanks for you advices. I thus created a new replica profile : {     "rule_id": 2,     "rule_name": "replicated3over2rooms",     "ruleset": 2,     "type": 1,     "min_size": 3,     "max_size": 4,     "steps": [         {             "op": "take",             "item": -1,             "item_name": "default"         },         {             "op": "choose_firstn",             "num": 0,             "type": "room"         },         {             "op": "chooseleaf_firstn",             "num": 2,             "type": "host"         }, # et c'est fini         {             "op": "emit"         }     ] } It works well. Now I am concerned by the pool in erasure coding. The point is that it's the data pool for cephfs (the metadata is in replica 3 and now replicated over our two rooms). For now, the data pool for cephfs is in *erasure coding k=3, m=2* (at the creation of the cluster we had only 5 osd servers). As noticed befors by Paul Emmerich, this cannot be redundantly splitted over 2 rooms (as 3 chunks are required to reconstruct the datas). Now, we have 6 OSD servers, and soon it will be 7, thus I was thinking to create a new pool (eg. k=4, m=2 or k=3, m=3) and a rule to split the chunks over our 2 rooms and to use this new pool as cache tier to migrate softly all the datas from the old pool to the new one. But according to https://documentation.suse.com/ses/6/html/ses-all/ceph-pools.html#pool-migra... "You can use the cache tier method to migrate from a replicated pool to either an erasure coded or another replicated pool. Migrating from an erasure coded pool is not supported." Warning: You Cannot Migrate RBD Images and CephFS Exports to an EC Pool You cannot migrate RBD images and CephFS exports from a replicated pool to an EC pool. EC pools can store data but not metadata. The header object of the RBD will fail to be flushed. The same applies for CephFS. Thus my question is *how can I migrate a data pool in EC of a cephfs to another EC pool ?* Thanks for your advices. F. Le 03/12/2019 à 04:07, Konstantin Shalygin a écrit :

On 12/2/19 5:56 PM, Francois Legrand wrote:

...

For replica, what is the best way to change crush profile ? Is it to create a new replica profile, and set this profile as crush rulest for the pool (something like ceph osd pool set {pool-name} crush_ruleset my_new_rule) ?

Indeed. Then you can delete/do what you want with old crush rule.

k

Thanks again for your answer. I still have few questions before going on. It seems that some metadata should remains on the original data pool, preventing it's deletion (http://ceph.com/geen-categorie/ceph-pool-migration/ and https://www.spinics.net/lists/ceph-users/msg41374.html). Thus does doing a cp and then a rm of the original files (instead of mv) allows to get rid of the remaining metadata in the original data pool ? Is it then possible to remove the original pool after migration (and how, because I guess that I have to define before the default location for data to the new pool) ? How snapshots are affected (do I have to remove all of them before the operation) ? Happy new year. F. Le 24/12/2019 à 03:53, Konstantin Shalygin a écrit :

On 12/19/19 10:22 PM, Francois Legrand wrote:

...

Thus my question is *how can I migrate a data pool in EC of a cephfs to another EC pool ?*

I suggest this:

# create you new ec pool

# `ceph osd pool application enable ec_new cephfs`

# `ceph fs add_data_pool cephfs ec_new`

# `setfattr -n ceph.dir.layout -v pool=ec_new /cephfs/ec_migration`

And then copy your content via userland tools.

k

I don't want to remove cephfs_meta pool but cephfs_datapool. To be clear : I have now cephfs consisting of a cephfs_metapool and a cephfs_datapool. I want to add a new data pool cephfs_datapool2, migrate all data from cephfs_datapool to cephfs_datapool2 and then remove the original cephfs_datapool. My goal is to end with a cephfs with cephfs_meta and cephfs_datapool2 (i.e replace the original cephfs_datapool by cephfs_datapool2). But from what I've seen, there should be also some "metadata"  in the cephfs_datapool (it sounds weird to me) which should remains after moving objects and prevent its deletion. F. Le 14/01/2020 à 07:54, Konstantin Shalygin a écrit :

On 1/6/20 5:50 PM, Francois Legrand wrote:

...

I still have few questions before going on. It seems that some metadata should remains on the original data pool, preventing it's deletion (http://ceph.com/geen-categorie/ceph-pool-migration/ and https://www.spinics.net/lists/ceph-users/msg41374.html). Thus does doing a cp and then a rm of the original files (instead of mv) allows to get rid of the remaining metadata in the original data pool ? Is it then possible to remove the original pool after migration (and how, because I guess that I have to define before the default location for data to the new pool) ? How snapshots are affected (do I have to remove all of them before the operation) ?

Why you need to remove cephfs_meta pool?

k