Re: Re layout help: need chassis local io to minimize net links
What does “traffic” mean? Reads? Writes will have to hit the net regardless of any machinations.
On Jun 29, 2020, at 7:31 PM, Harry G. Coin <hgcoin@gmail.com> wrote:
I need exactly what ceph is for a whole lot of work, that work just doesn't represent a large fraction of the total local traffic. Ceph is the right choice. Plainly ceph has tremendous support for replication within a chassis, among chassis and among racks. I just need intra-chassis traffic to not hit the net much. Seems not such an unreasonable thing given the intra-chassis crush rules and all. After all.. ceph's name wasn't chosen for where it can't go....
On 6/29/20 1:57 PM, Marc Roos wrote: I wonder if you should not have chosen a different product? Ceph is meant to distribute data across nodes, racks, data centers etc. For a nail use a hammer, for a screw use a screw driver. -----Original Message----- To: ceph-users@ceph.io Subject: *****SPAM***** [ceph-users] layout help: need chassis local io to minimize net links Hi I have a few servers each with 6 or more disks, with a storage workload that's around 80% done entirely within each server. From a work-to-be-done perspective there's no need for 80% of the load to traverse network interfaces, the rest needs what ceph is all about. So I cooked up a set of crush maps and pools, one map/pool for each server and one map/pool for the whole. Skipping the long story, the performance remains network link speed bound and has got to change. "Chassis local" io is too slow. I even tried putting a mon within each server. I'd like to avoid having to revert to some other HA filesystem per server with ceph at the chassis layer if I can help it. Any notions that would allow 'chassis local' rbd traffic to avoid or mostly avoid leaving the box? Thanks!
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the thinking. By 'traffic' I mean: when a user space rbd write has as a destination three replica osds in the same chassis, does the whole write get shipped out to the mon and then back, or just the write metadata to the mon, with the actual write data content not having to cross a physical ethernet cable but directly to the chassis-local osds via the 'virtual' internal switch? I thought when I read the layout of how ceph works only the control traffic goes to the mons, the data directly from the generator to the osds. Did I get that wrong? All the 'usual suspects' like lossy ethernets and miswirings, etc. have been checked. It's actually painful to sit and wait while 'update-initramfs' can take over a minute when the vm is chassis-local to the osds getting the write info. On 6/29/20 9:55 PM, Anthony D'Atri wrote:
What does “traffic” mean? Reads? Writes will have to hit the net regardless of any machinations.
On Jun 29, 2020, at 7:31 PM, Harry G. Coin <hgcoin@gmail.com> wrote:
I need exactly what ceph is for a whole lot of work, that work just doesn't represent a large fraction of the total local traffic. Ceph is the right choice. Plainly ceph has tremendous support for replication within a chassis, among chassis and among racks. I just need intra-chassis traffic to not hit the net much. Seems not such an unreasonable thing given the intra-chassis crush rules and all. After all.. ceph's name wasn't chosen for where it can't go....
On 6/29/20 1:57 PM, Marc Roos wrote: I wonder if you should not have chosen a different product? Ceph is meant to distribute data across nodes, racks, data centers etc. For a nail use a hammer, for a screw use a screw driver. -----Original Message----- To: ceph-users@ceph.io Subject: *****SPAM***** [ceph-users] layout help: need chassis local io to minimize net links Hi I have a few servers each with 6 or more disks, with a storage workload that's around 80% done entirely within each server. From a work-to-be-done perspective there's no need for 80% of the load to traverse network interfaces, the rest needs what ceph is all about. So I cooked up a set of crush maps and pools, one map/pool for each server and one map/pool for the whole. Skipping the long story, the performance remains network link speed bound and has got to change. "Chassis local" io is too slow. I even tried putting a mon within each server. I'd like to avoid having to revert to some other HA filesystem per server with ceph at the chassis layer if I can help it. Any notions that would allow 'chassis local' rbd traffic to avoid or mostly avoid leaving the box? Thanks!
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the thinking. By 'traffic' I mean: when a user space rbd write has as a destination three replica osds in the same chassis
eek.
does the whole write get shipped out to the mon and then back
Mons are control-plane only.
All the 'usual suspects' like lossy ethernets and miswirings, etc. have been checked. It's actually painful to sit and wait while 'update-initramfs' can take over a minute when the vm is chassis-local to the osds getting the write info.
You have shared almost none of your hardware or use-case. We know that you’re doing convergence, with unspecified CPU, memory, drives. We also don’t know how heavy your colocated compute workload is. Since you mention update-initramfs, I’ll guess that your workload is VMs with RBD volumes attached to libvirt/QEMU? With unspecified RBD cache configuration. We also know nothing of your network setup and saturation. I have to suspect that either you’re doing something fundamentally wrong, or should just set up a RAID6 volume and carve out LVMs.
Anthony asked about the 'use case'. Well, I haven't gone into details because I worried it wouldn't help much. From a 'ceph' perspective, the sandbox layout goes like this: 4 pretty much identical old servers, each with 6 drives, and a smaller server just running a mon to break ties. Usual front-side lan, separate back-side networking setup. Each of the servers is running a few vms, all more or less identical for the test case. Each of the vms is supported by a rbd via user space libvirt (not kernel mapped). Each rbd belongs to a pool that is entirely local to the chassis, presently a replica on 3 of the osds. One of the littler vms runs a mon+mgr per chassis. Of course what's important is there's a pool that spans the chassis and does all the usual things for userland ceph is good at. But for these tests I just unplugged all that. So, do any process that involves a bunch of little writes -- like installing a package or updating a initramfs and be ready to sit for a long time. All the drives are 7200 rpm SATA spinners. CPU's are not overloaded (fewer vms than cores), no swapping, memory left over. All write-back caching, virtio drives. Ceph octopus latest, though it's no better than nautilus performance wise in this case. Ubuntu LTS/focal/20.04 I think. Checked all the networking stats, no dropped packets, no overflow buffers and anyhow there shouldn't be any important traffic on the front side and only ceph owns the back end. No ceph problems reported, all pgs active, nothing misplaced, no erasure coded pools. So, there's a tiny novel, thanks for sticking with it! On 6/29/20 11:12 PM, Anthony D'Atri wrote:
Thanks for the thinking. By 'traffic' I mean: when a user space rbd write has as a destination three replica osds in the same chassis eek.
does the whole write get shipped out to the mon and then back Mons are control-plane only.
All the 'usual suspects' like lossy ethernets and miswirings, etc. have been checked. It's actually painful to sit and wait while 'update-initramfs' can take over a minute when the vm is chassis-local to the osds getting the write info. You have shared almost none of your hardware or use-case. We know that you’re doing convergence, with unspecified CPU, memory, drives. We also don’t know how heavy your colocated compute workload is. Since you mention update-initramfs, I’ll guess that your workload is VMs with RBD volumes attached to libvirt/QEMU? With unspecified RBD cache configuration. We also know nothing of your network setup and saturation.
I have to suspect that either you’re doing something fundamentally wrong, or should just set up a RAID6 volume and carve out LVMs. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Anthony D'Atri
-
Harry G. Coin