OK, where to start. I have been debugging intensively the last two days,
but can't seem to wrap my head around the performance issues we see in
one of our two hyperconverged (ceph) proxmox clusters.
Let me introduce our two clusters and some of the debugging results.
*1. Cluster for internal purposes (performs as expected)*
3 x Supermicro servers with identical specs:
CPU: 1 x 7-7700K CPU @ 4.20GHz (1 Socket) 4 cores / 8 threads
RAM: 64 GB RAM
OSDs: 4 per node. 1 per SSD (Intel S4610) (12 OSDs in all)
1 x 10GbE RJ45 nic. MTU 9000 No bonding
_A total of 3 servers with a total of 12 OSDs_
Network:
1 x Unifi Switch 16 XG
*2. Cluster for VPS's for customers (performs much worse than internal)*
3 x Dell R630 with the following specs:
CPU: 2 x E5-2697 v3 @ 2.60GHz (2 Sockets) 28 cores / 56 threads
RAM 256GB
OSDs: 10 per node. 1 per SSD (Intel S4610)
1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy
layer3+4). MTU 9000
2 x Supermicro X11SRM-VF with the following specs:
CPU 1 x 1 W-2145 CPU @ 3.70GHz (1 Socket) 8 cores / 16 threads
RAM: 256 GB
OSDs 8 per node. 1 per SSD (Intel S4610)
1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy
layer3+4). MTU 9000
1 x Dell R630 with the following specs:
CPU 2 x CPU E5-2696 v4 @ 2.20GHz (2 Sockets) 44 cores / 88 threads
RAM: 256 GB
OSDs 8 per node. 1 per SSD (Intel S4610)
1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy
layer3+4). MTU 9000
_A total of 6 servers with a total of 54 OSDs_
Network:
2 x Dell N4032F 10GbE SFP+ Switch connected with MLAG. Each node is
connected to each switch.
To get a fair comparison i made the following fio tests one one host in
each cluster on a rbd block device that i created:
*1. cluster:*
|fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1
--gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64
--size=4G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][100.0%][w=25.0MiB/s][w=6409 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=869177: Wed Nov 13 11:15:59 2019
write: IOPS=4158, BW=16.2MiB/s (17.0MB/s)(4096MiB/252126msec); 0 zone resets
bw ( KiB/s): min= 2075, max=32968, per=99.96%, avg=16627.60,
stdev=9635.42, samples=504
iops : min= 518, max= 8242, avg=4156.88, stdev=2408.86, samples=504
cpu : usr=0.53%, sys=3.81%, ctx=109599, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s),
io=4096MiB (4295MB), run=252126-252126msec
Disk stats (read/write):
rbd0: ios=46/1221898, merge=0/1870438, ticks=25/4654920,
in_queue=1980016, util=84.70%|
*2. cluster*
|fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1
--gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64
--size=4G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][99.9%][w=7024KiB/s][w=1756 IOPS][eta 00m:01s]
test: (groupid=0, jobs=1): err= 0: pid=794096: Wed Nov 13 11:25:56 2019
write: IOPS=1353, BW=5415KiB/s (5545kB/s)(4096MiB/774601msec); 0 zone resets
bw ( KiB/s): min= 40, max=30600, per=100.00%, avg=5420.24,
stdev=3710.17, samples=1547
iops : min= 10, max= 7650, avg=1355.06, stdev=927.54, samples=1547
cpu : usr=0.16%, sys=1.19%, ctx=100028, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=5415KiB/s (5545kB/s), 5415KiB/s-5415KiB/s (5545kB/s-5545kB/s),
io=4096MiB (4295MB), run=774601-774601msec
Disk stats (read/write):
rbd0: ios=0/1222639, merge=0/1784089, ticks=0/12124812,
in_queue=9514280, util=45.14%|
And identical rados bench tests:
1. cluster:https://i.imgur.com/AdARCA6.png
2. clusterhttps://i.imgur.com/Di7mYQh.png
I have fio tested all disks. I have tested the network. I can't seem to
find the reason why performance on my 2. cluster is relatively poor
compared to 1. cluster.
--
Dennis Højgaard
Powerhosting Support
*t:* +45 7222 4457 |*e:* dh(a)powerhosting.dk |*w:* https://powerhosting.dk
Hi Folks
I'm writing a backup/sync process for our ceph cluster. The process
takes a snapshot of the system that's being backed up and rsync's from
that to another ceph cluster
I was hoping to use the snapshot xattrs to verify a successful backup by
comparing ceph.dir.rbytes and ceph.dir.rentries between the snapshot and
backup target.
However, while files added to the source directory after the snapshot is
taken don't appear in the snapshot, ceph.dir.rbytes and ceph.dir.rctime
are updated in the xattrs of both.
$ fallocate -l 1K test1
$ getfattr -n ceph.dir.rbytes .
# file: .
ceph.dir.rbytes="1024"
$ mkdir .snap/testsnap
$ fallocate -l 2K test2
$ getfattr -n ceph.dir.rbytes . .snap/testsnap
# file: .
ceph.dir.rbytes="3072"
# file: .snap/testsnap
ceph.dir.rbytes="3072"
$ getfattr -n ceph.dir.rctime . .snap/testsnap
# file: .
ceph.dir.rctime="1573558448.09736131125"
# file: .snap/testsnap
ceph.dir.rctime="1573558448.09736131125"
$ tree -a --si . .snap
.
|-- [1.0k] test1
`-- [2.0k] test2
.snap
`-- [3.1k] testsnap
`-- [1.0k] test1
1 directory, 3 files
Have I misunderstood? Is this expected behaviour?
Cheers
Toby
--
Toby Darling, Scientific Computing (2N249)
MRC Laboratory of Molecular Biology
Francis Crick Avenue
Cambridge Biomedical Campus
Cambridge CB2 0QH
Phone 01223 267070
diesel home delivery bangalore: Get Diesel fuel to your generators and machines with no restrictions on minimum order and save money by avoiding fuel theft. Home delivery of fuel in Bangalore is simple, call 8884144444 or place order online.
https://powerbaba.com
BONJOUR! Speakeng India Provides Best French Language Classes in Bangalore,With Our Certified Experts & Get 100% Assistance.
http://www.speakengindia.com/french/
The GlossnGlass are the best bridal makeup artist in bangalore go beyond just creating immaculate makeup looks. We also train and produce the finest makeup artists through our professional makeup artist courses.With a team of expert trainers from the makeup industry, our makeup academy offers various intensive professional and personal grooming courses.
https://www.glossnglass.com/
<a href=https://www.abbssm.edu.in>Bangalore b schools</a>:best pgdm college in bangalore: Welcome to your future. We are the difference that makes you special. This is not just an institution, it's a springboard, and it's a catalyst. At ABBS, you will challenge yourself to learn, develop and r4e-engineer yourself to meet the demands of the country and the world. You are the future, and it is our responsibility to nurture the future.The MBA/PGDM at ABBS is specifically designed to prepare graduates in the emerging markets around the globe. The course is a transformative journey-offering unparalleled opportunity along with access to the best global management knowledge, corporate internships and placements from the finest companies in the market.this is one of the top 10 pgdm colleges in bangalore.
