On 19/03/2021 19:41, Stefan Kooman wrote:
On 3/19/21 7:20 PM, Andrej Filipcic wrote:
Hi,
I am testing 15.2.10 on a large cluster (RH8). cephfs pool (size=1) with 122 nvme OSDs works fine till the number of clients is relatively low. Writing from 400 kernel clients (ior benchmark), 8 streams each, causes issues. Writes are initially fast at 100GB/s but then they drop to <1GB/s after few minutes, while the OSDs use 300% cpu each. My guess is that the OSDs are overloaded with requests from too many clients, since it does not happen till there are ~3-4 streams OSD. The OSDs log do not show anything problematic.
tried to increase osd_op_num_threads_per_shard_ssd, did not help. Restarting OSDs recovers the situation for few minutes.
Writing to HDD pool with 1500 HDDs does not have any issues at all under same conditions.
Any hints, settings to improve this?
Not yet, just questions. How many PG's per NVMe do you have? How much memory per OSD (osd_memory_target) is configured? Do you have enabled "bluefs_buffered_io" on the OSDs?
8k PGs/112 , so 67. memory target is 7GB, 256 GB memory /26 OSDs, about 1/2 of memory is free. nvme OSDs use 3-4GB for that test. bluefs_buffered_io is set to true. Cheers, Andrej
Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic@ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------