Hoping someone may be able to help point out where my bottleneck(s) may be. I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt. To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories. My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
15.41G 79% 4.36MB/s 0:56:09 (xfr#23165, ir-chk=4061/27259)
avg-cpu: %user %nice %system %iowait %steal %idle 0.17 0.00 1.34 13.23 0.00 85.26
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz aqu-sz %util rbd0 124.00 0.66 0.00 0.00 17.30 5.48 50.00 0.17 0.00 0.00 31.70 3.49 0.00 0.00 0.00 0.00 0.00 0.00 3.39 96.40
Rsync progress and iostat (during the rsync) from the rbd to a local ssd, to remove any bottlenecks doubling back to cephfs. About 16G in 1h, not exactly blazing, this being 5 of the 7000 directories I'm looking to offload to cephfs. Currently running 15.2.11, and the host is Ubuntu 20.04 (5.4.0-72-generic) with a single E5-2620, 64GB of memory, and 4x10GbT bond talking to ceph, iperf proves it out. EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here. Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends going to 512, but was hoping to avoid the performance overhead of the PG splits if possible, given perf is bad enough as is. Examining the main directory structure it looks like there are 7000 files per directory, about 60% of which are <1MiB, and in all totaling nearly 5GiB per directory. My fstab for this is:
xfs _netdev,noatime 0 0
I tried to increase the read_ahead_kb to 4M from 128K at /sys/block/rbd0/queue/read_ahead_kb to match the object/stripe size of the EC pool, but that doesn't appear to have had much of an impact. The only thing I can think of that I could possibly try as a change would be to increase the queue depth in the rbdmap up from 128, so thats my next bullet to fire. Attaching xfs_info in case there are any useful nuggets:
meta-data=/dev/rbd0 isize=256 agcount=81, agsize=268435455 blks = sectsz=512 attr=2, projid32bit=0 = crc=0 finobt=0, sparse=0, rmapbt=0 = reflink=0 data = bsize=4096 blocks=21483470848, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=0 log =internal log bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0
And rbd-info:
rbd image 'rbd-image-name: size 85 TiB in 22282240 objects order 22 (4 MiB objects) snapshot_count: 0 id: a09cac2b772af5 data_pool: rbd-ec82-pool block_name_prefix: rbd_data.29.a09cac2b772af5 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Mon Apr 12 18:44:38 2021 access_timestamp: Mon Apr 12 18:44:38 2021 modify_timestamp: Mon Apr 12 18:44:38 2021
Any other ideas or hints are greatly appreciated. Thanks, Reed
On Thu, May 27, 2021 at 02:54:00PM -0500, Reed Dier wrote:
Hoping someone may be able to help point out where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
(Disclaimer: no experience with cephfs) I found rsync a wonderful tool for long distances and large files, less so for local networks and small files, even with local disks. Usually I do something like ( cd src/ && tar --acls --xattrs --numeric-owner --sparse -cf - . ) | pv -pterab | (cd dst/ && tar --acls --xattrs --numeric-owner --sparse -xf -) If src and dst are not mounted on the same machine you can use netcat/socat to stream the tar from one system to the other, or pipe it through ssh if you need encrypted transport. This does not have the resume capability of rsync, but for small files it is much faster. After that you can still throw in a final rsync for changes accumulated while the initial transfer was running. Matthias
I had it on my list of things to possibly try, a tar in | tar out copy to see if it yielded different results. On its face, it seems like cp -a is getting ever so slightly better speed, but not a clear night and day difference. I will definitely look into this and report back any findings, positive or negative. Thanks for the suggestion, Reed
On May 28, 2021, at 3:24 AM, Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Thu, May 27, 2021 at 02:54:00PM -0500, Reed Dier wrote:
Hoping someone may be able to help point out where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
(Disclaimer: no experience with cephfs)
I found rsync a wonderful tool for long distances and large files, less so for local networks and small files, even with local disks.
Usually I do something like
( cd src/ && tar --acls --xattrs --numeric-owner --sparse -cf - . ) | pv -pterab | (cd dst/ && tar --acls --xattrs --numeric-owner --sparse -xf -)
If src and dst are not mounted on the same machine you can use netcat/socat to stream the tar from one system to the other, or pipe it through ssh if you need encrypted transport.
This does not have the resume capability of rsync, but for small files it is much faster. After that you can still throw in a final rsync for changes accumulated while the initial transfer was running.
Matthias
There is also a longstanding belief that using cpio saves you context switches and data through a pipe. ymmv.
On May 28, 2021, at 7:26 AM, Reed Dier <reed.dier@focusvq.com> wrote:
I had it on my list of things to possibly try, a tar in | tar out copy to see if it yielded different results.
On its face, it seems like cp -a is getting ever so slightly better speed, but not a clear night and day difference.
I will definitely look into this and report back any findings, positive or negative.
Thanks for the suggestion,
Reed
On May 28, 2021, at 3:24 AM, Matthias Ferdinand <mf+ml.ceph@mfedv.net> wrote:
On Thu, May 27, 2021 at 02:54:00PM -0500, Reed Dier wrote:
Hoping someone may be able to help point out where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
(Disclaimer: no experience with cephfs)
I found rsync a wonderful tool for long distances and large files, less so for local networks and small files, even with local disks.
Usually I do something like
( cd src/ && tar --acls --xattrs --numeric-owner --sparse -cf - . ) | pv -pterab | (cd dst/ && tar --acls --xattrs --numeric-owner --sparse -xf -)
If src and dst are not mounted on the same machine you can use netcat/socat to stream the tar from one system to the other, or pipe it through ssh if you need encrypted transport.
This does not have the resume capability of rsync, but for small files it is much faster. After that you can still throw in a final rsync for changes accumulated while the initial transfer was running.
Matthias
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Reed, Have you tried just start multiple rsync process simultaneously to transfer different directories? Distributed system like ceph often benefits from more parallelism. Weiwen Hu
在 2021年5月28日，03:54，Reed Dier <reed.dier@focusvq.com> 写道：
Hoping someone may be able to help point out where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
15.41G 79% 4.36MB/s 0:56:09 (xfr#23165, ir-chk=4061/27259)
avg-cpu: %user %nice %system %iowait %steal %idle 0.17 0.00 1.34 13.23 0.00 85.26
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz aqu-sz %util rbd0 124.00 0.66 0.00 0.00 17.30 5.48 50.00 0.17 0.00 0.00 31.70 3.49 0.00 0.00 0.00 0.00 0.00 0.00 3.39 96.40
Rsync progress and iostat (during the rsync) from the rbd to a local ssd, to remove any bottlenecks doubling back to cephfs. About 16G in 1h, not exactly blazing, this being 5 of the 7000 directories I'm looking to offload to cephfs.
Currently running 15.2.11, and the host is Ubuntu 20.04 (5.4.0-72-generic) with a single E5-2620, 64GB of memory, and 4x10GbT bond talking to ceph, iperf proves it out. EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here. Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends going to 512, but was hoping to avoid the performance overhead of the PG splits if possible, given perf is bad enough as is.
Examining the main directory structure it looks like there are 7000 files per directory, about 60% of which are <1MiB, and in all totaling nearly 5GiB per directory.
My fstab for this is:
xfs _netdev,noatime 0 0
I tried to increase the read_ahead_kb to 4M from 128K at /sys/block/rbd0/queue/read_ahead_kb to match the object/stripe size of the EC pool, but that doesn't appear to have had much of an impact.
The only thing I can think of that I could possibly try as a change would be to increase the queue depth in the rbdmap up from 128, so thats my next bullet to fire.
Attaching xfs_info in case there are any useful nuggets:
meta-data=/dev/rbd0 isize=256 agcount=81, agsize=268435455 blks = sectsz=512 attr=2, projid32bit=0 = crc=0 finobt=0, sparse=0, rmapbt=0 = reflink=0 data = bsize=4096 blocks=21483470848, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=0 log =internal log bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0
And rbd-info:
rbd image 'rbd-image-name: size 85 TiB in 22282240 objects order 22 (4 MiB objects) snapshot_count: 0 id: a09cac2b772af5 data_pool: rbd-ec82-pool block_name_prefix: rbd_data.29.a09cac2b772af5 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Mon Apr 12 18:44:38 2021 access_timestamp: Mon Apr 12 18:44:38 2021 modify_timestamp: Mon Apr 12 18:44:38 2021
Any other ideas or hints are greatly appreciated.
Thanks, Reed
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Matthias Ferdinand
-
Reed Dier
-
胡 玮文