Thanks Ilya, Yeah maybe it should be made more explicit in the doc, but from the user point of view it's odd to have this limitation. Regarding the kubernetes-csi-addons that enable to do VolumeReplication , promote, demote, etc directly from Kubernetes it seems very nice and I did not know this project. https://github.com/csi-addons/kubernetes-csi-addons However, we are required to avoid "beta" feature from production environment that is why journal based mirroring is not used for our kubernetes volumes. This kubernetes-csi-addons seems to be very young project : version 0.12.0 and only 21 stars in github. I am not confident to use that in production. Are you aware of production deployments using it ? Regards Le jeu. 19 juin 2025 à 00:10, Ilya Dryomov <idryomov@gmail.com> a écrit :
On Wed, Jun 18, 2025 at 7:29 PM Julien Laurenceau <julien.laurenceau@pepitedata.com> wrote:
Hi,
I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode.
if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored.
if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster.
Reading the documentation
https://docs.ceph.com/en/squid/rbd/rbd-mirroring/,
I think that the problem is that the default mode for this command is "journal":
rbd mirror image enable {pool-name}/{image-name} {mode}
But, when enabling mirror on the pool there is no way to define mode=journal/snapshot:
rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode}
with mode=pool (default) or image.
Then I think when I do : rbd mirror pool enable ${mypool} image
I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK.
But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK.
In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode
Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode
Could someone please confirm, that this is not supported ?
Hi Julien,
Yes, snapshot-based mirroring is possible only in "image" mode. The "pool" mode is limited to journal-based mirroring. It's stated in the documentation, but perhaps not explicitly enough:
You can also mirror all images within a given pool or namespace when using journal-based mirroring.
What is the best workaround ? Cron cephadm shell script to enable mirror
on
all images ?
PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.
If you are using Ceph CSI, it would be much better to drive mirroring through Ceph CSI -- see VolumeReplication and related resources.
Thanks,
Ilya