Out of curiosity, when you chose EC over replication how did you weigh IOPS vs space amplification in your decision making process? I'm wondering if we should prioritize EC latency vs other tasks in future tuning efforts (it's always a tradeoff deciding what to focus on). Thanks, Mark On 10/22/19 2:35 AM, Frank Schilder wrote:
Getting decent RBD performance is not a trivial exercise. While at a first glance 61 SSDs for 245 clients sounds more or less OK, it does come down to a bit more than that.
The first thing is, how to get SSD performance out of SSDs with ceph. This post will provide very good clues and might already point out the bottleneck: https://yourcmc.ru/wiki/index.php?title=Ceph_performance . Do you have good enterprise SSDs?
Next thing to look at, what kind of data pool, replicated or erasure coded? If erasure coded, has the profile been benchmarked? There are very poor choices. Good ones are 4+m, 8+m. 4+m better IOps, 8+m better throughput. m>=2.
More complications: do you need to deploy more than one OSD per SSD to boost performance? This is indicated by the iodepth required in an fio benchmark to get full IOPs. Good SSDs deliver already spec performance with 1 OSD. More common ones require 2-4 OSDs per disk. Are you using ceph-volume already, its default is 2 OSDs per SSD (batch mode).
To give a base line, after extensive testing and working through all the required tuning steps, I could run about 250 VMs on a 6+2 EC data pool on 33 enterprise SAS SSDs with 1 OSD per disk, each VM getting 50IOPs write performance. This is probably what you would like to see as well.
If you use replicated data pool, this should be relatively easy. With EC data pool, this is a bit of a battle.
Good luck,
================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: ceph-users <ceph-users-bounces@lists.ceph.com> on behalf of Void Star Nill <void.star.nill@gmail.com> Sent: 22 October 2019 03:00 To: ceph-users Subject: [ceph-users] Fwd: large concurrent rbd operations block for over 15 mins!
Apparently the graph is too big, so my last post is stuck. Resending without the graph.
Thanks
---------- Forwarded message --------- From: Void Star Nill <void.star.nill@gmail.com<mailto:void.star.nill@gmail.com>> Date: Mon, Oct 21, 2019 at 4:41 PM Subject: large concurrent rbd operations block for over 15 mins! To: ceph-users <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>>
Hello,
I have been running some benchmark tests with a mid-size cluster and I am seeing some issues. Wanted to know if this is a bug or something that can be tuned. Appreciate any help on this.
- I have a 15 node Ceph cluster, with 3 monitors and 12 data nodes with total 61 OSDs on SSDs running 14.2.4 nautilus (stable) version. Each node has 100G link. - I have 245 client machines from which I am triggering rbd operations. Each client has 25G link - rbd operations include, creating an RBD image of 50G size and layering feature, mapping the image to the client machine, formatting the device in ext4 format, mounting it, running dd to write to the full disk and cleaning up (unmount, unmap and remove).
If I run these RBD operations concurrently on a small number of machines (say 16-20), they run very well and I see good throughput. All image operations (except for dd) take less than 2 seconds.
However, when I scale it up to 245 clients, each running these operations concurrently, I see lot of operations getting hung for a long time and the overall throughput reduces drastically.
For example, some of the format operations take over 10-15 mins!!!
Note that, all operations do complete - so its most likely not a deadlock kind of situation.
I dont see any errors in ceph.log on the monitor nodes. However, the clients do report "hung_task_timeout" in dmesg logs.
As you can see in the below image, half the format operations are completing in less than a second time, while the other half is over 10mins (y axis is in seconds)
[11117.113618] INFO: task umount:9902 blocked for more than 120 seconds. [11117.113677] Tainted: G OE 4.15.0-51-generic #55~16.04.1-Ubuntu [11117.113731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [11117.113787] umount D 0 9902 9901 0x00000000 [11117.113793] Call Trace: [11117.113804] __schedule+0x3d6/0x8b0 [11117.113810] ? _raw_spin_unlock_bh+0x1e/0x20 [11117.113814] schedule+0x36/0x80 [11117.113821] wb_wait_for_completion+0x64/0x90 [11117.113828] ? wait_woken+0x80/0x80 [11117.113831] __writeback_inodes_sb_nr+0x8e/0xb0 [11117.113835] writeback_inodes_sb+0x27/0x30 [11117.113840] __sync_filesystem+0x51/0x60 [11117.113844] sync_filesystem+0x26/0x40 [11117.113850] generic_shutdown_super+0x27/0x120 [11117.113854] kill_block_super+0x2c/0x80 [11117.113858] deactivate_locked_super+0x48/0x80 [11117.113862] deactivate_super+0x5a/0x60 [11117.113866] cleanup_mnt+0x3f/0x80 [11117.113868] __cleanup_mnt+0x12/0x20 [11117.113874] task_work_run+0x8a/0xb0 [11117.113881] exit_to_usermode_loop+0xc4/0xd0 [11117.113885] do_syscall_64+0x100/0x130 [11117.113887] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [11117.113891] RIP: 0033:0x7f0094384487 [11117.113893] RSP: 002b:00007fff4199efc8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 [11117.113897] RAX: 0000000000000000 RBX: 0000000000944030 RCX: 00007f0094384487 [11117.113899] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 0000000000944210 [11117.113900] RBP: 0000000000944210 R08: 0000000000000000 R09: 0000000000000014 [11117.113902] R10: 00000000000006b2 R11: 0000000000000246 R12: 00007f009488d83c [11117.113903] R13: 0000000000000000 R14: 0000000000000000 R15: 00007fff4199f250 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io