Re: ceph fs crashes on simple fio test
On Tue, Sep 10, 2019 at 1:11 PM Frank Schilder <frans@dtu.dk> wrote:
Hi Robert,
I have meta data on SSD (3xrep) and data on 8+2 EC on spinning disks, so the speed difference is orders of magnitudes. Our usage is quite meta data heavy, so this suits us well. In particular since EC pools are high throughput with large IO sizes.
As long as one uses fio with direct=1 (probably also if using sync=1 and/or fsync=1), everything is fine and behaves as you describe. IOPs fluctuate but adjust to media speed. No problems at all.
As mentioned in my last update (I cut it out below), the destructive fio command runs with direct=0 and neither sync=1 nor fsync=1. This test just writes as fast as it can (to buffers) without waiting for acks. I would expect that a ceph client would translate that to synced or direct IO, which would be fine.
But it doesn't. Instead, it pushes the IO also as fast as possible to the cluster. I have seen 40kops write on the EC pool (on 100+ HDDs) that can handle maybe 1kops write in total. The queues were constantly increasing at an incredible rate (several hundred ops per second). I hope with the change of cut_off=high that heartbeats will not get lost any more, but this will still destabilize our ceph cluster quite dramatically.
Changing the cut_off to high will not allow heartbeats to not get lost (heartbeats have a priority far above the high mark). What cut_off = high does is put replication ops into the main queue instead of the strict priority queue. That way an OSD doesn't get DDOSed from it's peers and is never able to service it's own clients. When I did my fio testing, was on FireFly/Hammer and on RBD, so I can't talk specifically to newer versions and CephFS. We haven't had time to set up our test cluster, so I can't run benches at the moment.
My problem is not so much that such an IO pattern could occur in reasonable software, but - that someone might try just for fun, and that - the number of 500+ clients might occasionally produce such a workload by aggregation.
I find it somewhat alarming that a storage system that promises data integrity and reliability can be taken down with a publicly available benchmark tool in a matter of a few dozen seconds by ordinary users. Potentially with damaging effects. I guess something similar could be achieved with a modified rogue client.
I would expect that a storage cluster should have basic self-defence mechanisms that prevent this kind of overload or DOS attack by throttling clients with crazy IO requests. Are there any settings that can be enabled to prevent this from happening?
---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
Dear all, I found a partial solution to the problem and I also repeated a bit of testing, see below. # Client-sided solution, works for single-client IO The hard solution is to mount cephfs with the option "sync". This will translate any IO to direct IO and successfully throttle clients no matter how they perform IO. This will even work in multi-client set-ups. A somewhat less restrictive option is to set low values for vm.dirty_[background_]bytes to allow some buffered IO for small bursts. I tried with vm.dirty_background_bytes = 524288 vm.dirty_bytes = 1048576 and less restrictive vm.dirty_background_bytes = 2097152 vm.dirty_bytes = 67108864 (without sync mount option) and it seems to have the desired effect. It is possible to obtain good large-IO size throughput while limiting small IO size IOPs to a healthy level. Of course, this does not address destructive multi-client IO patterns, which must be addressed on the server side. # Test observations Today I repeated a shorter test to avoid crashing the cluster bad. We are in production and I don't have a test cluster. Therefore, if anyone could try this on a test cluster and check if the observations can be confirmed, that would be great. Here is a one-line command: fio -name=rand-write -directory=/mnt/cephfs/home/frans/fio -filename_format=tmp/fio-\$jobname-\$jobnum-\$filenum -rw=randwrite -bs=4K -numjobs=4 -time_based=1 -runtime=5 -filesize=100G -ioengine=sync -direct=0 -iodepth=1 Adjust runtime and numjobs to increasingly higher values to increase stress. In my original tests I observed OSD outages with numjobs=4 and runtime=30 already. Note that these occur several minutes after the fio command completes. Here are my today's observations with "osd_op_queue=wpq" and "osd_op_queue_cut_off=high" and a 5 sec run time: - High IOPs (>4kops) on the data pool come in two waves. - The first wave does not cause slow ops. - There is a phase of low activity. - A second wave starts and now slow meta data ops are reported by the MDS. Health level becomes warn. - The cluster crunches through the meta data ops for a minute or so and then settles. This is quite a long time considering a 5 secs burst. - OSDs did not go out, but this could be due to not running the test long enough. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
participants (2)
-
Frank Schilder
-
Robert LeBlanc