ceph rbd mirror snapshot-based does not work if mirror-mode is pool and not image
Hi, I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode. if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored. if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster. Reading the documentation https://docs.ceph.com/en/squid/rbd/rbd-mirroring/, I think that the problem is that the default mode for this command is "journal": rbd mirror image enable {pool-name}/{image-name} {mode} But, when enabling mirror on the pool there is no way to define mode=journal/snapshot: rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode} with mode=pool (default) or image. Then I think when I do : rbd mirror pool enable ${mypool} image I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK. But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK. In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode Could someone please confirm, that this is not supported ? What is the best workaround ? Cron cephadm shell script to enable mirror on all images ? PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.
On Wed, Jun 18, 2025 at 7:29 PM Julien Laurenceau <julien.laurenceau@pepitedata.com> wrote:
Hi,
I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode.
if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored.
if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster.
Reading the documentation https://docs.ceph.com/en/squid/rbd/rbd-mirroring/, I think that the problem is that the default mode for this command is "journal":
rbd mirror image enable {pool-name}/{image-name} {mode}
But, when enabling mirror on the pool there is no way to define mode=journal/snapshot:
rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode}
with mode=pool (default) or image.
Then I think when I do : rbd mirror pool enable ${mypool} image
I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK.
But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK.
In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode
Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode
Could someone please confirm, that this is not supported ?
Hi Julien, Yes, snapshot-based mirroring is possible only in "image" mode. The "pool" mode is limited to journal-based mirroring. It's stated in the documentation, but perhaps not explicitly enough: You can also mirror all images within a given pool or namespace when using journal-based mirroring.
What is the best workaround ? Cron cephadm shell script to enable mirror on all images ?
PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.
If you are using Ceph CSI, it would be much better to drive mirroring through Ceph CSI -- see VolumeReplication and related resources. Thanks, Ilya
Thanks Ilya, Yeah maybe it should be made more explicit in the doc, but from the user point of view it's odd to have this limitation. Regarding the kubernetes-csi-addons that enable to do VolumeReplication , promote, demote, etc directly from Kubernetes it seems very nice and I did not know this project. https://github.com/csi-addons/kubernetes-csi-addons However, we are required to avoid "beta" feature from production environment that is why journal based mirroring is not used for our kubernetes volumes. This kubernetes-csi-addons seems to be very young project : version 0.12.0 and only 21 stars in github. I am not confident to use that in production. Are you aware of production deployments using it ? Regards Le jeu. 19 juin 2025 à 00:10, Ilya Dryomov <idryomov@gmail.com> a écrit :
On Wed, Jun 18, 2025 at 7:29 PM Julien Laurenceau <julien.laurenceau@pepitedata.com> wrote:
Hi,
I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode.
if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored.
if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster.
Reading the documentation
https://docs.ceph.com/en/squid/rbd/rbd-mirroring/,
I think that the problem is that the default mode for this command is "journal":
rbd mirror image enable {pool-name}/{image-name} {mode}
But, when enabling mirror on the pool there is no way to define mode=journal/snapshot:
rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode}
with mode=pool (default) or image.
Then I think when I do : rbd mirror pool enable ${mypool} image
I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK.
But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK.
In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode
Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode
Could someone please confirm, that this is not supported ?
Hi Julien,
Yes, snapshot-based mirroring is possible only in "image" mode. The "pool" mode is limited to journal-based mirroring. It's stated in the documentation, but perhaps not explicitly enough:
You can also mirror all images within a given pool or namespace when using journal-based mirroring.
What is the best workaround ? Cron cephadm shell script to enable mirror
on
all images ?
PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.
If you are using Ceph CSI, it would be much better to drive mirroring through Ceph CSI -- see VolumeReplication and related resources.
Thanks,
Ilya
On Thu, Jun 19, 2025 at 9:54 AM Julien Laurenceau <julien.laurenceau@pepitedata.com> wrote:
Thanks Ilya,
Yeah maybe it should be made more explicit in the doc, but from the user point of view it's odd to have this limitation.
Regarding the kubernetes-csi-addons that enable to do VolumeReplication , promote, demote, etc directly from Kubernetes it seems very nice and I did not know this project. https://github.com/csi-addons/kubernetes-csi-addons However, we are required to avoid "beta" feature from production environment that is why journal based mirroring is not used for our kubernetes volumes. This kubernetes-csi-addons seems to be very young project : version 0.12.0 and only 21 stars in github. I am not confident to use that in production. Are you aware of production deployments using it ?
I don't have a reference for someone using it directly, but this project is being used as a foundation of a downstream product shipped by Red Hat/IBM. It's fare to say that snapshot-based mirroring in standalone RBD is more "beta" (i.e. has more rough edges, etc) than when used from within Kubernetes thanks to the additional automation and safe guards on the Ceph CSI side. Thanks, Ilya
participants (2)
-
Ilya Dryomov
-
Julien Laurenceau