I have a Ceph instance I'm wherein I'm trying to use erasure coding with RBD (and libvirt). I've followed https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codin... and enabled overwrites so that RBD can be used. In doing so, I've set the data_pool to the erasure coded pool, with set to the replicated pool. I have the following in ceph.conf: rbd_default_data_pool = libvirt-pool-ec Here's an rbd info on the image I've created (notice the "data_pool" config): rbd image 'xxxxxxx': size 500 GiB in 128000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 7e37e442592b data_pool: libvirt-pool-ec block_name_prefix: rbd_data.6.7e37e442592b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Fri Sep 13 16:34:06 2024 access_timestamp: Fri Sep 13 16:34:06 2024 modify_timestamp: Fri Sep 13 16:34:06 2024 The problem, when I attach this rbd image to a VM, I cannot write to it at all, I get an I/O error. But, when I create and rbd image without the "rbd_default_data_pool = libvirt-pool-ec" setting, I can write fine. Wanted to see if anyone has any ideas, maybe I'm missing something simple. Thanks in advance!
Here You have guide https://febryandana.xyz/posts/deploy-ceph-openstack-cluster/ in short ceph osd pool create images 128 ceph osd pool set images size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done ceph osd pool create volumes 128 ceph osd pool set volumes size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done ceph osd pool create vms 128 ceph osd pool set vms size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done ceph osd erasure-code-profile set ec-22-profile k=2 m=2 crush-device-class=ssd ceph osd erasure-code-profile ls ceph osd erasure-code-profile get ec-22-profile ceph osd pool create images_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done ceph osd pool create volumes_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done ceph osd pool create vms_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done ceph osd pool ls detail ceph osd pool set images_data allow_ec_overwrites true ceph osd pool set volumes_data allow_ec_overwrites true ceph osd pool set vms_data allow_ec_overwrites true ceph osd pool application enable volumes rbd ceph osd pool application enable images rbd ceph osd pool application enable vms rbd ceph osd pool application enable volumes_data rbd ceph osd pool application enable images_data rbd ceph osd pool application enable vms_data rbd On ceph.conf You need put as below [client.glance] rbd default data pool = images_data [client.cinder] rbd default data pool = volumes_data [client.nova] rbd default data pool = vms_data for permission you probably also need add caps mon = "allow r, allow command \\"osd blacklist\\", allow command \\"osd blocklist\\", allow command \\"blacklistop\\", allow command \\"blocklistop\\"" Newer versions might not work with blacklist anymore.
first comment on the replicated pools: the replication size for rbd pools of 2 is not suitable for production clusters. It is only a matter of time before you lose data. Joachim www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Am Sa., 14. Sept. 2024 um 14:04 Uhr schrieb <przemek.kuczynski@gmail.com>:
Here You have guide
https://febryandana.xyz/posts/deploy-ceph-openstack-cluster/
in short
ceph osd pool create images 128 ceph osd pool set images size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create volumes 128 ceph osd pool set volumes size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create vms 128 ceph osd pool set vms size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd erasure-code-profile set ec-22-profile k=2 m=2 crush-device-class=ssd ceph osd erasure-code-profile ls ceph osd erasure-code-profile get ec-22-profile
ceph osd pool create images_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create volumes_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create vms_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool ls detail
ceph osd pool set images_data allow_ec_overwrites true ceph osd pool set volumes_data allow_ec_overwrites true ceph osd pool set vms_data allow_ec_overwrites true
ceph osd pool application enable volumes rbd ceph osd pool application enable images rbd ceph osd pool application enable vms rbd ceph osd pool application enable volumes_data rbd ceph osd pool application enable images_data rbd ceph osd pool application enable vms_data rbd On ceph.conf You need put as below
[client.glance] rbd default data pool = images_data
[client.cinder] rbd default data pool = volumes_data
[client.nova] rbd default data pool = vms_data for permission you probably also need add
caps mon = "allow r, allow command \\"osd blacklist\\", allow command \\"osd blocklist\\", allow command \\"blacklistop\\", allow command \\"blocklistop\\"" Newer versions might not work with blacklist anymore. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
100% agree. I’ve seen claims that SSDs synch quickly so 2 is enough. Such claims are shortsighted. I have personally witnessed cases of device failures and OSD flaps / crashes at just the wrong times resulted in no clear latest copy of PGs. There are cases where data loss isn’t catastrophic, but unless you’re sure, you want either R3 for durability or EC to minimize space amp. A 2,2 profile, for example.
On Sep 15, 2024, at 10:35 AM, Joachim Kraftmayer <joachim.kraftmayer@clyso.com> wrote:
first comment on the replicated pools: the replication size for rbd pools of 2 is not suitable for production clusters. It is only a matter of time before you lose data. Joachim
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Am Sa., 14. Sept. 2024 um 14:04 Uhr schrieb <przemek.kuczynski@gmail.com>:
Here You have guide
https://febryandana.xyz/posts/deploy-ceph-openstack-cluster/
in short
ceph osd pool create images 128 ceph osd pool set images size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create volumes 128 ceph osd pool set volumes size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create vms 128 ceph osd pool set vms size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd erasure-code-profile set ec-22-profile k=2 m=2 crush-device-class=ssd ceph osd erasure-code-profile ls ceph osd erasure-code-profile get ec-22-profile
ceph osd pool create images_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create volumes_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create vms_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool ls detail
ceph osd pool set images_data allow_ec_overwrites true ceph osd pool set volumes_data allow_ec_overwrites true ceph osd pool set vms_data allow_ec_overwrites true
ceph osd pool application enable volumes rbd ceph osd pool application enable images rbd ceph osd pool application enable vms rbd ceph osd pool application enable volumes_data rbd ceph osd pool application enable images_data rbd ceph osd pool application enable vms_data rbd On ceph.conf You need put as below
[client.glance] rbd default data pool = images_data
[client.cinder] rbd default data pool = volumes_data
[client.nova] rbd default data pool = vms_data for permission you probably also need add
caps mon = "allow r, allow command \\"osd blacklist\\", allow command \\"osd blocklist\\", allow command \\"blacklistop\\", allow command \\"blocklistop\\"" Newer versions might not work with blacklist anymore. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
As a reminder, there's this one waiting ;-) https://tracker.ceph.com/issues/66641 Frédéric. PS: For the record, Andre's problem was related to the 'caps' (https://www.reddit.com/r/ceph/comments/1ffzfjc/ceph_rbd_werasure_coding/) ----- Le 15 Sep 24, à 18:02, Anthony D'Atri anthony.datri@gmail.com a écrit :
100% agree.
I’ve seen claims that SSDs synch quickly so 2 is enough. Such claims are shortsighted.
I have personally witnessed cases of device failures and OSD flaps / crashes at just the wrong times resulted in no clear latest copy of PGs.
There are cases where data loss isn’t catastrophic, but unless you’re sure, you want either R3 for durability or EC to minimize space amp. A 2,2 profile, for example.
On Sep 15, 2024, at 10:35 AM, Joachim Kraftmayer <joachim.kraftmayer@clyso.com> wrote:
first comment on the replicated pools: the replication size for rbd pools of 2 is not suitable for production clusters. It is only a matter of time before you lose data. Joachim
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306
Am Sa., 14. Sept. 2024 um 14:04 Uhr schrieb <przemek.kuczynski@gmail.com>:
Here You have guide
https://febryandana.xyz/posts/deploy-ceph-openstack-cluster/
in short
ceph osd pool create images 128 ceph osd pool set images size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create volumes 128 ceph osd pool set volumes size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create vms 128 ceph osd pool set vms size 2 while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd erasure-code-profile set ec-22-profile k=2 m=2 crush-device-class=ssd ceph osd erasure-code-profile ls ceph osd erasure-code-profile get ec-22-profile
ceph osd pool create images_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create volumes_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool create vms_data 128 128 erasure ec-22-profile while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done
ceph osd pool ls detail
ceph osd pool set images_data allow_ec_overwrites true ceph osd pool set volumes_data allow_ec_overwrites true ceph osd pool set vms_data allow_ec_overwrites true
ceph osd pool application enable volumes rbd ceph osd pool application enable images rbd ceph osd pool application enable vms rbd ceph osd pool application enable volumes_data rbd ceph osd pool application enable images_data rbd ceph osd pool application enable vms_data rbd On ceph.conf You need put as below
[client.glance] rbd default data pool = images_data
[client.cinder] rbd default data pool = volumes_data
[client.nova] rbd default data pool = vms_data for permission you probably also need add
caps mon = "allow r, allow command \\"osd blacklist\\", allow command \\"osd blocklist\\", allow command \\"blacklistop\\", allow command \\"blocklistop\\"" Newer versions might not work with blacklist anymore. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, Can you run "ceph osd pool get <<your_ec_pool_here>> allow_ec_overwrites"? What is the output of that? Thank you, Bogdan Velica On Fri, Sep 13, 2024 at 8:46 PM <andre@drenet.net> wrote:
I have a Ceph instance I'm wherein I'm trying to use erasure coding with RBD (and libvirt). I've followed
https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codin... and enabled overwrites so that RBD can be used. In doing so, I've set the data_pool to the erasure coded pool, with set to the replicated pool.
I have the following in ceph.conf:
rbd_default_data_pool = libvirt-pool-ec
Here's an rbd info on the image I've created (notice the "data_pool" config):
rbd image 'xxxxxxx': size 500 GiB in 128000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 7e37e442592b data_pool: libvirt-pool-ec block_name_prefix: rbd_data.6.7e37e442592b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Fri Sep 13 16:34:06 2024 access_timestamp: Fri Sep 13 16:34:06 2024 modify_timestamp: Fri Sep 13 16:34:06 2024 The problem, when I attach this rbd image to a VM, I cannot write to it at all, I get an I/O error. But, when I create and rbd image without the "rbd_default_data_pool = libvirt-pool-ec" setting, I can write fine.
Wanted to see if anyone has any ideas, maybe I'm missing something simple. Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I did have "allow_ec_overwrites" set, which is why I was stumped. The resolution in my case was editing my client.libvirt user adding to the mon caps, and allowing the erasure coded pool on the osd caps: ceph auth caps client.libvirt mon 'allow r, allow command "osd blacklist", allow command "osd blocklist", allow command "blacklistop", allow command "blocklistop"' osd 'allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool, allow rwx pool=images, allow rwx pool=libvirt-pool-ec' On 2024-09-13 15:56, Bogdan Adrian Velica wrote:
Hi,
Can you run "ceph osd pool get <<your_ec_pool_here>> allow_ec_overwrites"? What is the output of that?
Thank you, Bogdan Velica
On Fri, Sep 13, 2024 at 8:46 PM <andre@drenet.net> wrote:
I have a Ceph instance I'm wherein I'm trying to use erasure coding with RBD (and libvirt). I've followed
https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codin...
and enabled overwrites so that RBD can be used. In doing so, I've set the data_pool to the erasure coded pool, with set to the replicated pool.
I have the following in ceph.conf:
rbd_default_data_pool = libvirt-pool-ec
Here's an rbd info on the image I've created (notice the "data_pool"
config):
rbd image 'xxxxxxx': size 500 GiB in 128000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 7e37e442592b data_pool: libvirt-pool-ec block_name_prefix: rbd_data.6.7e37e442592b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Fri Sep 13 16:34:06 2024 access_timestamp: Fri Sep 13 16:34:06 2024 modify_timestamp: Fri Sep 13 16:34:06 2024 The problem, when I attach this rbd image to a VM, I cannot write to it at all, I get an I/O error. But, when I create and rbd image without the "rbd_default_data_pool = libvirt-pool-ec" setting, I can write fine.
Wanted to see if anyone has any ideas, maybe I'm missing something simple. Thanks in advance! _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
andre＠drenet.net
-
Anthony D'Atri
-
Bogdan Adrian Velica
-
Frédéric Nass
-
Joachim Kraftmayer
-
przemek.kuczynski＠gmail.com