Hi, Thank you for any time. We are tracking some very slow pwrite64() calls to a ceph filesystem - 20965 11:04:24.049186 <... pwrite64 resumed>) = 65536 <4.489594> 20966 11:04:24.069765 <... pwrite64 resumed>) = 65536 <4.508859> 20967 11:04:24.090354 <... pwrite64 resumed>) = 65536 <4.510256> But other pwrite64()s from the same program in other threads to other files on the same ceph fs seem fine; we cannot really reproduce this, but it happens occasionally. It seems we are spending some time in ceph_aio_write() when this is happening (see call graph below) I've noticed THP (transparent Huge Pages) is enabled. We are running version 15.2.17 on CentOS 7.9 Do not seem to be under any significant memory pressure when this happens, just many threads of this app blocked on i/o in pwrite64()s. I am suggesting an upgrade, but until then, do you think this situation involves ceph and could be improved if we disable THP ? Thanks for any advice or suggestion, -mark Call graph of app when slow pwrites64()s are happening - --87.08%--system_call_fastpath |--58.20%--sys_pwrite64 | --58.03%--vfs_write | do_sync_write | ceph_aio_write | |--54.33%--generic_file_buffered_write | | |--27.09%--ceph_write_begin | | | |--17.01%--grab_cache_page_write_begin | | | | |--7.51%--add_to_page_cache_lru | | | | | |--4.28%--__add_to_page_cache_locked | | | | | | --3.95%--mem_cgroup_cache_charge | | | | | | mem_cgroup_charge_common | | | | | | --3.75%--__mem_cgroup_commit_charge | | | | | --3.23%--lru_cache_add | | | | | __lru_cache_add | | | | | --2.94%--pagevec_lru_move_fn | | | | | |--0.70%--mem_cgroup_page_lruvec | | | | | |--0.67%--__pagevec_lru_add_fn | | | | | --0.57%--release_pages | | | | |--5.31%--__page_cache_alloc | | | | | --5.00%--alloc_pages_current | | | | | __alloc_pages_nodemask | | | | | --4.23%--get_page_from_freelist | | | | | |--1.85%--__rmqueue | | | | | | --1.49%--list_del | | | | | | __list_del_entry | | | | | --1.74%--list_del | | | | | __list_del_entry | | | | --3.92%--__find_lock_page | | | | __find_get_page | | | | --3.46%--radix_tree_lookup_slot | | | | |--2.76%--radix_tree_descend | | | | --0.70%--__radix_tree_lookup | | | | radix_tree_descend | | | --9.45%--ceph_update_writeable_page | | | --8.94%--readpage_nounlock | | | --8.60%--ceph_osdc_readpages | | | --8.18%--submit_request | | | __submit_request | | | calc_target.isra.50 | | | ceph_pg_to_up_acting_osds | | | crush_do_rule | | | crush_choose_firstn | | | |--4.89%--crush_choose_firstn | | | | is_out.isra.2.part.3 | | | --3.30%--crush_bucket_choose | | |--14.50%--copy_user_enhanced_fast_string | | |--6.34%--ceph_write_end | | | |--3.76%--set_page_dirty | | | | ceph_set_page_dirty | | | | --2.86%--__set_page_dirty_nobuffers | | | | |--0.92%--_raw_spin_unlock_irqrestore | | | | --0.59%--radix_tree_tag_set | | | --1.88%--unlock_page | | | __wake_up_bit | | |--3.44%--iov_iter_fault_in_readable | | --2.70%--mark_page_accessed | |--2.54%--mutex_lock | | __mutex_lock_slowpath | | --2.14%--schedule_preempt_disabled | | __schedule | | | | | |--0.82%--finish_task_switch | | | __perf_event_task_sched_in | | | perf_pmu_enable | | | x86_pmu_enable | | | --0.80%--intel_pmu_enable_all | | | --0.74%--__intel_pmu_enable_all.isra.23 | | | --0.69%--native_write_msr_safe | | --0.75%--__perf_event_task_sched_out | --0.56%--mutex_unlock | __mutex_unlock_slowpath ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.