27 May
2021
27 May
'21
9:54 p.m.
Hoping someone may be able to help point out where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that.
This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid
array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of
small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just
unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
15.41G 79% 4.36MB/s 0:56:09
(xfr#23165, ir-chk=4061/27259)
avg-cpu: %user %nice %system %iowait %steal
%idle
0.17 0.00 1.34 13.23 0.00 85.26
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s
wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz aqu-sz
%util
rbd0 124.00 0.66 0.00 0.00 17.30 5.48 50.00 0.17
0.00 0.00 31.70 3.49 0.00 0.00 0.00 0.00 0.00 0.00 3.39
96.40
Rsync progress and iostat (during the rsync) from the rbd to a local ssd, to remove any
bottlenecks doubling back to cephfs.
About 16G in 1h, not exactly blazing, this being 5 of the 7000 directories I'm looking
to offload to cephfs.
Currently running 15.2.11, and the host is Ubuntu 20.04 (5.4.0-72-generic) with a single
E5-2620, 64GB of memory, and 4x10GbT bond talking to ceph, iperf proves it out.
EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other
216 being 2TB 7.2K SATA. So there are quite a few spindles in play here.
Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends
going to 512, but was hoping to avoid the performance overhead of the PG splits if
possible, given perf is bad enough as is.
Examining the main directory structure it looks like there are 7000 files per directory,
about 60% of which are <1MiB, and in all totaling nearly 5GiB per directory.
My fstab for this is:
xfs _netdev,noatime 0 0
I tried to increase the read_ahead_kb to 4M from 128K at
/sys/block/rbd0/queue/read_ahead_kb to match the object/stripe size of the EC pool, but
that doesn't appear to have had much of an impact.
The only thing I can think of that I could possibly try as a change would be to increase
the queue depth in the rbdmap up from 128, so thats my next bullet to fire.
Attaching xfs_info in case there are any useful nuggets:
meta-data=/dev/rbd0 isize=256
agcount=81, agsize=268435455 blks
= sectsz=512 attr=2, projid32bit=0
= crc=0 finobt=0, sparse=0, rmapbt=0
= reflink=0
data = bsize=4096 blocks=21483470848, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=0
log =internal log bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
And rbd-info:
rbd image 'rbd-image-name:
size 85 TiB in 22282240 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: a09cac2b772af5
data_pool: rbd-ec82-pool
block_name_prefix: rbd_data.29.a09cac2b772af5
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten,
data-pool
op_features:
flags:
create_timestamp: Mon Apr 12 18:44:38 2021
access_timestamp: Mon Apr 12 18:44:38 2021
modify_timestamp: Mon Apr 12 18:44:38 2021
Any other ideas or hints are greatly appreciated.
Thanks,
Reed
28 May
28 May
10:24 a.m.
On Thu, May 27, 2021 at 02:54:00PM -0500, Reed Dier wrote:
Hoping someone may be able to help point out where my
bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that.
This was not an ideal scenario, rather it was a rescue mission to dump a large, aging
raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of
small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just
unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
(Disclaimer: no experience with cephfs)
I found rsync a wonderful tool for long distances and large files, less
so for local networks and small files, even with local disks.
Usually I do something like
( cd src/ && tar --acls --xattrs --numeric-owner --sparse -cf - . ) |
pv -pterab |
(cd dst/ && tar --acls --xattrs --numeric-owner --sparse -xf -)
If src and dst are not mounted on the same machine you can use
netcat/socat to stream the tar from one system to the other, or pipe it
through ssh if you need encrypted transport.
This does not have the resume capability of rsync, but for small files
it is much faster. After that you can still throw in a final rsync for
changes accumulated while the initial transfer was running.
Matthias
4:26 p.m.
I had it on my list of things to possibly try, a tar in | tar out copy to see if it
yielded different results.
On its face, it seems like cp -a is getting ever so slightly better speed, but not a clear
night and day difference.
I will definitely look into this and report back any findings, positive or negative.
Thanks for the suggestion,
Reed
On May 28, 2021, at 3:24 AM, Matthias Ferdinand
<mf+ml.ceph(a)mfedv.net> wrote:
On Thu, May 27, 2021 at 02:54:00PM -0500, Reed Dier wrote:
Hoping someone may be able to help point out
where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that.
This was not an ideal scenario, rather it was a rescue mission to dump a large, aging
raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of
small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just
unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
(Disclaimer: no experience with cephfs)
I found rsync a wonderful tool for long distances and large files, less
so for local networks and small files, even with local disks.
Usually I do something like
( cd src/ && tar --acls --xattrs --numeric-owner --sparse -cf - . ) |
pv -pterab |
(cd dst/ && tar --acls --xattrs --numeric-owner --sparse -xf -)
If src and dst are not mounted on the same machine you can use
netcat/socat to stream the tar from one system to the other, or pipe it
through ssh if you need encrypted transport.
This does not have the resume capability of rsync, but for small files
it is much faster. After that you can still throw in a final rsync for
changes accumulated while the initial transfer was running.
Matthias 
5:03 p.m.
There is also a longstanding belief that using cpio saves you context switches and data
through a pipe. ymmv.
On May 28, 2021, at 7:26 AM, Reed Dier
<reed.dier(a)focusvq.com> wrote:
I had it on my list of things to possibly try, a tar in | tar out copy to see if it
yielded different results.
On its face, it seems like cp -a is getting ever so slightly better speed, but not a
clear night and day difference.
I will definitely look into this and report back any findings, positive or negative.
Thanks for the suggestion,
Reed
On May 28, 2021, at 3:24 AM, Matthias Ferdinand
<mf+ml.ceph(a)mfedv.net> wrote:
On Thu, May 27, 2021 at 02:54:00PM -0500, Reed Dier wrote:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io Hoping someone may be able to help point out
where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that.
This was not an ideal scenario, rather it was a rescue mission to dump a large, aging
raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of
small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just
unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
(Disclaimer: no experience with cephfs)
I found rsync a wonderful tool for long distances and large files, less
so for local networks and small files, even with local disks.
Usually I do something like
( cd src/ && tar --acls --xattrs --numeric-owner --sparse -cf - . ) |
pv -pterab |
(cd dst/ && tar --acls --xattrs --numeric-owner --sparse -xf -)
If src and dst are not mounted on the same machine you can use
netcat/socat to stream the tar from one system to the other, or pipe it
through ssh if you need encrypted transport.
This does not have the resume capability of rsync, but for small files
it is much faster. After that you can still throw in a final rsync for
changes accumulated while the initial transfer was running.
Matthias 
1:03 p.m.
Hi Reed,
Have you tried just start multiple rsync process simultaneously to transfer different
directories? Distributed system like ceph often benefits from more parallelism.
Weiwen Hu
在 2021年5月28日，03:54，Reed Dier
<reed.dier(a)focusvq.com> 写道：
Hoping someone may be able to help point out where my bottleneck(s) may be.
I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that.
This was not an ideal scenario, rather it was a rescue mission to dump a large, aging
raid array before it was too late, so I'm working with the hand I was dealt.
To further conflate the issues, the main directory structure consists of lots and lots of
small file sizes, and deep directories.
My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just
unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal.
15.41G 79% 4.36MB/s 0:56:09
(xfr#23165, ir-chk=4061/27259)
avg-cpu: %user %nice %system %iowait %steal
%idle
0.17 0.00 1.34 13.23 0.00 85.26
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s
wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz aqu-sz
%util
rbd0 124.00 0.66 0.00 0.00 17.30 5.48 50.00 0.17
0.00 0.00 31.70 3.49 0.00 0.00 0.00 0.00 0.00 0.00 3.39
96.40
Rsync progress and iostat (during the rsync) from the rbd to a local ssd, to remove any
bottlenecks doubling back to cephfs.
About 16G in 1h, not exactly blazing, this being 5 of the 7000 directories I'm
looking to offload to cephfs.
Currently running 15.2.11, and the host is Ubuntu 20.04 (5.4.0-72-generic) with a single
E5-2620, 64GB of memory, and 4x10GbT bond talking to ceph, iperf proves it out.
EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the
other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here.
Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler
recommends going to 512, but was hoping to avoid the performance overhead of the PG splits
if possible, given perf is bad enough as is.
Examining the main directory structure it looks like there are 7000 files per directory,
about 60% of which are <1MiB, and in all totaling nearly 5GiB per directory.
My fstab for this is:
xfs _netdev,noatime 0 0
I tried to increase the read_ahead_kb to 4M from 128K at
/sys/block/rbd0/queue/read_ahead_kb to match the object/stripe size of the EC pool, but
that doesn't appear to have had much of an impact.
The only thing I can think of that I could possibly try as a change would be to increase
the queue depth in the rbdmap up from 128, so thats my next bullet to fire.
Attaching xfs_info in case there are any useful nuggets:
meta-data=/dev/rbd0 isize=256
agcount=81, agsize=268435455 blks
= sectsz=512 attr=2, projid32bit=0
= crc=0 finobt=0, sparse=0, rmapbt=0
= reflink=0
data = bsize=4096 blocks=21483470848, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=0
log =internal log bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
And rbd-info:
rbd image 'rbd-image-name:
size 85 TiB in 22282240 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: a09cac2b772af5
data_pool: rbd-ec82-pool
block_name_prefix: rbd_data.29.a09cac2b772af5
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten,
data-pool
op_features:
flags:
create_timestamp: Mon Apr 12 18:44:38 2021
access_timestamp: Mon Apr 12 18:44:38 2021
modify_timestamp: Mon Apr 12 18:44:38 2021
Any other ideas or hints are greatly appreciated.
Thanks,
Reed
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1061
days inactive
1062
days old
4 comments
4 participants
participants (4)
-
Anthony D'Atri -
Matthias Ferdinand -
Reed Dier -
胡 玮文