22 Jun
2021
22 Jun
'21
10:51 p.m.
Hello Cephers,
On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm migrating a 40
To image from a 3+2 EC pool to a 8+2 one.
The use case is Veeam backup on XFS filesystems, mounted via KRBD.
Backups are running, and I can see 200MB/s Throughput.
But my migration (rbd migrate prepare / execute) is staling at 4% for 6h now.
When the backups are not running, I can see a little 20MB/s of throughput,
certainly my migration.
I need a month to migrate 40 to at that speed !
As I use a KRBD client, I cannot remap the rbd image straight after the rbd
prepare. So the filesystem is not usable until the migration is completed.
Not really usable for me...
Is anyone has a clue either to speed up the rbd migration, or another method
to move/copy an image between 2 pools, with the minimum downtime ?
I thought of rbd export-diff | rbd import-diff, while mounted, and another
unmapped before switching...
But, it forces me to rename my image, because if I use another data pool, the
metadata pool stays the same.
Can you see another method ?
--
Gilles
23 Jun
23 Jun
1:12 p.m.
Le 2021-06-22 20:21, Gilles Mocellin a écrit :
Hello Cephers,
On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm
migrating a 40
To image from a 3+2 EC pool to a 8+2 one.
The use case is Veeam backup on XFS filesystems, mounted via KRBD.
Backups are running, and I can see 200MB/s Throughput.
But my migration (rbd migrate prepare / execute) is staling at 4% for
6h now.
When the backups are not running, I can see a little 20MB/s of
throughput,
certainly my migration.
I need a month to migrate 40 TB at that speed !
Hello,
It seems worst, this morning, I still see a 4% completed status.
The metrics I can see other the last 12h are not showing much
activity...
So it seems more stale than slow.
Is anyone using that rbd migration command, on reasonable sized image ?
And of course between different data pools ?
During my initial tests, it worked, but with little and quite empty test
images (1 TB)...
I will investigate an alternate method, like :
- snapshot src image
- copy src image to new data pool (new image name)
- unmount src image
- export-diff | import-diff delta from src image with snap to dest image
- test mount dest image
- delete src image
- rename dest image
The unknown duration here is for the export/import diffs. But during the
initial copy, my production can continue.
Anyon has already done something similar and can comment on the
duration, or pinpoints ?
--
Gilles
5:21 p.m.
вт, 22 июн. 2021 г. в 23:22, Gilles Mocellin <gilles.mocellin(a)nuagelibre.org>rg>:
Hello Cephers,
On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm migrating a 40
To image from a 3+2 EC pool to a 8+2 one.
The use case is Veeam backup on XFS filesystems, mounted via KRBD.
Backups are running, and I can see 200MB/s Throughput.
But my migration (rbd migrate prepare / execute) is staling at 4% for 6h now.
When the backups are not running, I can see a little 20MB/s of throughput,
certainly my migration.
I need a month to migrate 40 to at that speed !
As I use a KRBD client, I cannot remap the rbd image straight after the rbd
prepare. So the filesystem is not usable until the migration is completed.
Not really usable for me...
Is anyone has a clue either to speed up the rbd migration, or another method
to move/copy an image between 2 pools, with the minimum downtime ?
I thought of rbd export-diff | rbd import-diff, while mounted, and another
unmapped before switching...
But, it forces me to rename my image, because if I use another data pool, the
metadata pool stays the same.
Can you see another method ?
I suggest that you cancel the migration and don't ever attempt it
again because big EC setups are very easy to overload with IOPS.
When I worked at croit GmbH, we had a very unhappy customer with
almost the same setup as you are trying to achieve: Veeam Backup, XFS
on rbd on a 8+3 EC pool of HDDs. Their complaint was that both the
backup and restore were extremely slow, ~3 MB/s, and with 200 ms of
latency, but I would call their cluster overloaded due to too many
concurrent backups. We tried, unsuccessfully, to tune their setup, but
our final recommendation (successfully benchmarked but rejected due to
costs) was to create a separate replica 3 pool for new backups.
--
Alexander E. Patrakov
CV: http://u.pc.cd/wT8otalK
7:01 p.m.
Le 2021-06-23 14:51, Alexander E. Patrakov a écrit :
вт, 22 июн. 2021 г. в 23:22, Gilles Mocellin
<gilles.mocellin(a)nuagelibre.org>rg>:
>
> Hello Cephers,
>
> On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm
> migrating a 40
> To image from a 3+2 EC pool to a 8+2 one.
>
> The use case is Veeam backup on XFS filesystems, mounted via KRBD.
>
> Backups are running, and I can see 200MB/s Throughput.
>
> But my migration (rbd migrate prepare / execute) is staling at 4% for
> 6h now.
>
> When the backups are not running, I can see a little 20MB/s of
> throughput,
> certainly my migration.
[...]
I suggest that you cancel the migration and don't
ever attempt it
again because big EC setups are very easy to overload with IOPS.
When I worked at croit GmbH, we had a very unhappy customer with
almost the same setup as you are trying to achieve: Veeam Backup, XFS
on rbd on a 8+3 EC pool of HDDs. Their complaint was that both the
backup and restore were extremely slow, ~3 MB/s, and with 200 ms of
latency, but I would call their cluster overloaded due to too many
concurrent backups. We tried, unsuccessfully, to tune their setup, but
our final recommendation (successfully benchmarked but rejected due to
costs) was to create a separate replica 3 pool for new backups.
Argh...
The load was not terrible. And when there wasn't any backup, it was very
low, but the migration really seems blocked.
And now, more strange, the abort doesn't finish. The image status has
become unknown :
root -> rbd status veeam-repos/veeam-repo4-vol2
Watchers:
watcher=100.99.103.54:0/1497373484 client.5113986
cookie=139751457839168
Migration:
source: veeam-repos/veeam-repo4-vol2 (17e5e3267adad3)
destination: veeam-repos/veeam-repo4-vol2 (4debed1a2ed31b)
state: unknown
The connected client is the rbd abort command.
1056
days inactive
1057
days old
3 comments
2 participants
participants (2)
-
Alexander E. Patrakov -
Gilles Mocellin