Hi, I'm testing mirroring in ceph and dealt with very slow mirroring operation, Cluster A (primary) is on a Ceph hyperconverged on proxmox and cluster B (secondary) is a Ceph cluster managed by cephadm. These 2 have been peered via bootstrap. Moving 60 GiB VM to the mirrored pool in cluster A it took 5-6 hours to fully sync in the cluster B (SSD device only) Moving 60 GiB VM from cluster A to cluster B in replica 3 SSD pool took 2 minutes (SSD device only) It looks like for this purpose cluster wont be higher than 6-7 MiB/s, is this a normal behaviour ? Ceph -s during mirroring output and rbd info below ceph -s cluster: id: ID health: HEALTH_OK services: mon: 3 daemons, quorum r620-13-1,r620-13-9,r620-13-4 (age 12d) mgr: r620-13-7.zxzajo(active, since 24h), standbys: r620-13-10.wmzodp mds: 1/1 daemons up osd: 24 osds: 24 up (since 6d), 24 in (since 6d) rbd-mirror: 1 daemon active (1 hosts) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 16 pools, 1287 pgs objects: 151.21k objects, 551 GiB usage: 1.6 TiB used, 6.3 TiB / 7.9 TiB avail pgs: 1287 active+clean io: client: 15 KiB/s rd, 6.5 MiB/s wr, 11 op/s rd, 419 op/s wr -------------------------------------------- rbd info ceph_proxmox/vm-118-disk-0 rbd image 'vm-118-disk-0': size 60 GiB in 15360 objects order 22 (4 MiB objects) snapshot_count: 0 id: abe8c560f69498 block_name_prefix: rbd_data.abe8c560f69498 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Wed Oct 22 07:52:31 2025 access_timestamp: Wed Oct 22 07:52:31 2025 modify_timestamp: Wed Oct 22 07:52:31 2025 journal: abe8c560f69498 mirroring state: enabled mirroring mode: journal mirroring global id: id mirroring primary: false ----------------------------------------- In cluster A ceph.conf rbd_default_features = 125 Thanks ! Vivien
Hi, I haven't dealt with journal based mirroring in quite a while, but this section [0] in the docs seems relevant:
rbd-mirror tunables are set by default to values suitable for mirroring an entire pool. When using rbd-mirror to migrate single volumes between clusters you may achieve substantial performance gains by setting rbd_journal_max_payload_bytes=8388608 within the [client] config section of the local or centralized configuration. Note that this setting may allow rbd-mirror to present a substantial write workload to the destination cluster: monitor cluster performance closely during migrations and test carefully before running multiple migrations in parallel.
# ceph config help rbd_journal_max_payload_bytes rbd_journal_max_payload_bytes - maximum journal payload size before splitting (size, advanced) Default: 16384 Can update at runtime: true Services: [rbd] Regards, Eugen [0] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#enable-image-journaling-f... Zitat von "GLE, Vivien" <Vivien.GLE@inist.fr>:
Hi,
I'm testing mirroring in ceph and dealt with very slow mirroring operation,
Cluster A (primary) is on a Ceph hyperconverged on proxmox and cluster B (secondary) is a Ceph cluster managed by cephadm. These 2 have been peered via bootstrap.
Moving 60 GiB VM to the mirrored pool in cluster A it took 5-6 hours to fully sync in the cluster B (SSD device only)
Moving 60 GiB VM from cluster A to cluster B in replica 3 SSD pool took 2 minutes (SSD device only)
It looks like for this purpose cluster wont be higher than 6-7 MiB/s, is this a normal behaviour ?
Ceph -s during mirroring output and rbd info below
ceph -s cluster: id: ID health: HEALTH_OK
services: mon: 3 daemons, quorum r620-13-1,r620-13-9,r620-13-4 (age 12d) mgr: r620-13-7.zxzajo(active, since 24h), standbys: r620-13-10.wmzodp mds: 1/1 daemons up osd: 24 osds: 24 up (since 6d), 24 in (since 6d) rbd-mirror: 1 daemon active (1 hosts) rgw: 1 daemon active (1 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 16 pools, 1287 pgs objects: 151.21k objects, 551 GiB usage: 1.6 TiB used, 6.3 TiB / 7.9 TiB avail pgs: 1287 active+clean
io: client: 15 KiB/s rd, 6.5 MiB/s wr, 11 op/s rd, 419 op/s wr
--------------------------------------------
rbd info ceph_proxmox/vm-118-disk-0 rbd image 'vm-118-disk-0': size 60 GiB in 15360 objects order 22 (4 MiB objects) snapshot_count: 0 id: abe8c560f69498 block_name_prefix: rbd_data.abe8c560f69498 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Wed Oct 22 07:52:31 2025 access_timestamp: Wed Oct 22 07:52:31 2025 modify_timestamp: Wed Oct 22 07:52:31 2025 journal: abe8c560f69498 mirroring state: enabled mirroring mode: journal mirroring global id: id mirroring primary: false
-----------------------------------------
In cluster A ceph.conf
rbd_default_features = 125
Thanks !
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
Are the two clusters the same size? Pools both replicated? Do the nodes where rbd-mirror is running have the same CPU/RAM resources? Is one more >heavily loaded than the other?
Cluster has to be exactly the same to mirroring work well ?
rbd_mirror_journal_max_fetch_bytes and rbd_journal_max_payload_bytes=8388608
I didn't see this part in the docs, my bad. I added this to ceph.conf and it's working a lot faster One more thing: do you have more info about the Ceph subreddit ban, and if it will be lifted soon? Thanks Anthony and Eugen Vivien ________________________________ De : Eugen Block <eblock@nde.ag> Envoyé : jeudi 23 octobre 2025 07:58:08 À : ceph-users@ceph.io Objet : [ceph-users] Re: Very slow mirroring operation Hi, I haven't dealt with journal based mirroring in quite a while, but this section [0] in the docs seems relevant:
rbd-mirror tunables are set by default to values suitable for mirroring an entire pool. When using rbd-mirror to migrate single volumes between clusters you may achieve substantial performance gains by setting rbd_journal_max_payload_bytes=8388608 within the [client] config section of the local or centralized configuration. Note that this setting may allow rbd-mirror to present a substantial write workload to the destination cluster: monitor cluster performance closely during migrations and test carefully before running multiple migrations in parallel.
# ceph config help rbd_journal_max_payload_bytes rbd_journal_max_payload_bytes - maximum journal payload size before splitting (size, advanced) Default: 16384 Can update at runtime: true Services: [rbd] Regards, Eugen [0] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#enable-image-journaling-f... Zitat von "GLE, Vivien" <Vivien.GLE@inist.fr>:
Hi,
I'm testing mirroring in ceph and dealt with very slow mirroring operation,
Cluster A (primary) is on a Ceph hyperconverged on proxmox and cluster B (secondary) is a Ceph cluster managed by cephadm. These 2 have been peered via bootstrap.
Moving 60 GiB VM to the mirrored pool in cluster A it took 5-6 hours to fully sync in the cluster B (SSD device only)
Moving 60 GiB VM from cluster A to cluster B in replica 3 SSD pool took 2 minutes (SSD device only)
It looks like for this purpose cluster wont be higher than 6-7 MiB/s, is this a normal behaviour ?
Ceph -s during mirroring output and rbd info below
ceph -s cluster: id: ID health: HEALTH_OK
services: mon: 3 daemons, quorum r620-13-1,r620-13-9,r620-13-4 (age 12d) mgr: r620-13-7.zxzajo(active, since 24h), standbys: r620-13-10.wmzodp mds: 1/1 daemons up osd: 24 osds: 24 up (since 6d), 24 in (since 6d) rbd-mirror: 1 daemon active (1 hosts) rgw: 1 daemon active (1 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 16 pools, 1287 pgs objects: 151.21k objects, 551 GiB usage: 1.6 TiB used, 6.3 TiB / 7.9 TiB avail pgs: 1287 active+clean
io: client: 15 KiB/s rd, 6.5 MiB/s wr, 11 op/s rd, 419 op/s wr
--------------------------------------------
rbd info ceph_proxmox/vm-118-disk-0 rbd image 'vm-118-disk-0': size 60 GiB in 15360 objects order 22 (4 MiB objects) snapshot_count: 0 id: abe8c560f69498 block_name_prefix: rbd_data.abe8c560f69498 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Wed Oct 22 07:52:31 2025 access_timestamp: Wed Oct 22 07:52:31 2025 modify_timestamp: Wed Oct 22 07:52:31 2025 journal: abe8c560f69498 mirroring state: enabled mirroring mode: journal mirroring global id: id mirroring primary: false
-----------------------------------------
In cluster A ceph.conf
rbd_default_features = 125
Thanks !
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Are the two clusters the same size? Pools both replicated? Do the nodes where rbd-mirror is running have the same CPU/RAM resources? Is one more >heavily loaded than the other?
Cluster has to be exactly the same to mirroring work well ?
No, but I've seen issues where one has architecturally lower performance than the other. The rbd-mirror daemon running in the destination cluster is the one that shovels the bits around, so when moving in both directions, if the daemon on one side is constrained by saturated or slow CPU, misconfigured swap, etc. that could explain asymmetrical performance.
rbd_mirror_journal_max_fetch_bytes and rbd_journal_max_payload_bytes=8388608
I didn't see this part in the docs, my bad. I added this to ceph.conf and it's working a lot faster
Glad it helps. I got those suggestions from Jason Dilaman a handful of years ago when I was using rbd-mirror to migrate volumes between clusters for capacity management.
One more thing: do you have more info about the Ceph subreddit ban, and if it will be lifted soon?
No idea, but I'll ask our community manager if there's anything he can do.
Thanks Anthony and Eugen
Vivien
________________________________ De : Eugen Block <eblock@nde.ag> Envoyé : jeudi 23 octobre 2025 07:58:08 À : ceph-users@ceph.io Objet : [ceph-users] Re: Very slow mirroring operation
Hi,
I haven't dealt with journal based mirroring in quite a while, but this section [0] in the docs seems relevant:
rbd-mirror tunables are set by default to values suitable for mirroring an entire pool. When using rbd-mirror to migrate single volumes between clusters you may achieve substantial performance gains by setting rbd_journal_max_payload_bytes=8388608 within the [client] config section of the local or centralized configuration. Note that this setting may allow rbd-mirror to present a substantial write workload to the destination cluster: monitor cluster performance closely during migrations and test carefully before running multiple migrations in parallel.
# ceph config help rbd_journal_max_payload_bytes rbd_journal_max_payload_bytes - maximum journal payload size before splitting (size, advanced) Default: 16384 Can update at runtime: true Services: [rbd]
Regards, Eugen
[0] https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#enable-image-journaling-f...
Zitat von "GLE, Vivien" <Vivien.GLE@inist.fr>:
Hi,
I'm testing mirroring in ceph and dealt with very slow mirroring operation,
Cluster A (primary) is on a Ceph hyperconverged on proxmox and cluster B (secondary) is a Ceph cluster managed by cephadm. These 2 have been peered via bootstrap.
Moving 60 GiB VM to the mirrored pool in cluster A it took 5-6 hours to fully sync in the cluster B (SSD device only)
Moving 60 GiB VM from cluster A to cluster B in replica 3 SSD pool took 2 minutes (SSD device only)
It looks like for this purpose cluster wont be higher than 6-7 MiB/s, is this a normal behaviour ?
Ceph -s during mirroring output and rbd info below
ceph -s cluster: id: ID health: HEALTH_OK
services: mon: 3 daemons, quorum r620-13-1,r620-13-9,r620-13-4 (age 12d) mgr: r620-13-7.zxzajo(active, since 24h), standbys: r620-13-10.wmzodp mds: 1/1 daemons up osd: 24 osds: 24 up (since 6d), 24 in (since 6d) rbd-mirror: 1 daemon active (1 hosts) rgw: 1 daemon active (1 hosts, 1 zones)
data: volumes: 1/1 healthy pools: 16 pools, 1287 pgs objects: 151.21k objects, 551 GiB usage: 1.6 TiB used, 6.3 TiB / 7.9 TiB avail pgs: 1287 active+clean
io: client: 15 KiB/s rd, 6.5 MiB/s wr, 11 op/s rd, 419 op/s wr
--------------------------------------------
rbd info ceph_proxmox/vm-118-disk-0 rbd image 'vm-118-disk-0': size 60 GiB in 15360 objects order 22 (4 MiB objects) snapshot_count: 0 id: abe8c560f69498 block_name_prefix: rbd_data.abe8c560f69498 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Wed Oct 22 07:52:31 2025 access_timestamp: Wed Oct 22 07:52:31 2025 modify_timestamp: Wed Oct 22 07:52:31 2025 journal: abe8c560f69498 mirroring state: enabled mirroring mode: journal mirroring global id: id mirroring primary: false
-----------------------------------------
In cluster A ceph.conf
rbd_default_features = 125
Thanks !
Vivien
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Anthony D'Atri
-
Eugen Block
-
GLE, Vivien