RBD migration between 2 EC pools : very slow
Hello Cephers, On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm migrating a 40 To image from a 3+2 EC pool to a 8+2 one. The use case is Veeam backup on XFS filesystems, mounted via KRBD. Backups are running, and I can see 200MB/s Throughput. But my migration (rbd migrate prepare / execute) is staling at 4% for 6h now. When the backups are not running, I can see a little 20MB/s of throughput, certainly my migration. I need a month to migrate 40 to at that speed ! As I use a KRBD client, I cannot remap the rbd image straight after the rbd prepare. So the filesystem is not usable until the migration is completed. Not really usable for me... Is anyone has a clue either to speed up the rbd migration, or another method to move/copy an image between 2 pools, with the minimum downtime ? I thought of rbd export-diff | rbd import-diff, while mounted, and another unmapped before switching... But, it forces me to rename my image, because if I use another data pool, the metadata pool stays the same. Can you see another method ? -- Gilles
Le 2021-06-22 20:21, Gilles Mocellin a écrit :
Hello Cephers,
On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm migrating a 40 To image from a 3+2 EC pool to a 8+2 one.
The use case is Veeam backup on XFS filesystems, mounted via KRBD.
Backups are running, and I can see 200MB/s Throughput.
But my migration (rbd migrate prepare / execute) is staling at 4% for 6h now.
When the backups are not running, I can see a little 20MB/s of throughput, certainly my migration.
I need a month to migrate 40 TB at that speed !
Hello, It seems worst, this morning, I still see a 4% completed status. The metrics I can see other the last 12h are not showing much activity... So it seems more stale than slow. Is anyone using that rbd migration command, on reasonable sized image ? And of course between different data pools ? During my initial tests, it worked, but with little and quite empty test images (1 TB)... I will investigate an alternate method, like : - snapshot src image - copy src image to new data pool (new image name) - unmount src image - export-diff | import-diff delta from src image with snap to dest image - test mount dest image - delete src image - rename dest image The unknown duration here is for the export/import diffs. But during the initial copy, my production can continue. Anyon has already done something similar and can comment on the duration, or pinpoints ? -- Gilles
вт, 22 июн. 2021 г. в 23:22, Gilles Mocellin <gilles.mocellin@nuagelibre.org>:
Hello Cephers,
On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm migrating a 40 To image from a 3+2 EC pool to a 8+2 one.
The use case is Veeam backup on XFS filesystems, mounted via KRBD.
Backups are running, and I can see 200MB/s Throughput.
But my migration (rbd migrate prepare / execute) is staling at 4% for 6h now.
When the backups are not running, I can see a little 20MB/s of throughput, certainly my migration.
I need a month to migrate 40 to at that speed !
As I use a KRBD client, I cannot remap the rbd image straight after the rbd prepare. So the filesystem is not usable until the migration is completed.
Not really usable for me...
Is anyone has a clue either to speed up the rbd migration, or another method to move/copy an image between 2 pools, with the minimum downtime ?
I thought of rbd export-diff | rbd import-diff, while mounted, and another unmapped before switching...
But, it forces me to rename my image, because if I use another data pool, the metadata pool stays the same.
Can you see another method ?
I suggest that you cancel the migration and don't ever attempt it again because big EC setups are very easy to overload with IOPS. When I worked at croit GmbH, we had a very unhappy customer with almost the same setup as you are trying to achieve: Veeam Backup, XFS on rbd on a 8+3 EC pool of HDDs. Their complaint was that both the backup and restore were extremely slow, ~3 MB/s, and with 200 ms of latency, but I would call their cluster overloaded due to too many concurrent backups. We tried, unsuccessfully, to tune their setup, but our final recommendation (successfully benchmarked but rejected due to costs) was to create a separate replica 3 pool for new backups. -- Alexander E. Patrakov CV: http://u.pc.cd/wT8otalK
Le 2021-06-23 14:51, Alexander E. Patrakov a écrit :
вт, 22 июн. 2021 г. в 23:22, Gilles Mocellin <gilles.mocellin@nuagelibre.org>:
Hello Cephers,
On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm migrating a 40 To image from a 3+2 EC pool to a 8+2 one.
The use case is Veeam backup on XFS filesystems, mounted via KRBD.
Backups are running, and I can see 200MB/s Throughput.
But my migration (rbd migrate prepare / execute) is staling at 4% for 6h now.
When the backups are not running, I can see a little 20MB/s of throughput, certainly my migration.
[...]
I suggest that you cancel the migration and don't ever attempt it again because big EC setups are very easy to overload with IOPS.
When I worked at croit GmbH, we had a very unhappy customer with almost the same setup as you are trying to achieve: Veeam Backup, XFS on rbd on a 8+3 EC pool of HDDs. Their complaint was that both the backup and restore were extremely slow, ~3 MB/s, and with 200 ms of latency, but I would call their cluster overloaded due to too many concurrent backups. We tried, unsuccessfully, to tune their setup, but our final recommendation (successfully benchmarked but rejected due to costs) was to create a separate replica 3 pool for new backups.
Argh... The load was not terrible. And when there wasn't any backup, it was very low, but the migration really seems blocked. And now, more strange, the abort doesn't finish. The image status has become unknown : root -> rbd status veeam-repos/veeam-repo4-vol2 Watchers: watcher=100.99.103.54:0/1497373484 client.5113986 cookie=139751457839168 Migration: source: veeam-repos/veeam-repo4-vol2 (17e5e3267adad3) destination: veeam-repos/veeam-repo4-vol2 (4debed1a2ed31b) state: unknown The connected client is the rbd abort command.
participants (2)
-
Alexander E. Patrakov
-
Gilles Mocellin