When checking the RGW logs I can confirm that it is in fact the same issue as the one in the issue. 2023-09-20T12:52:06.670+0000 7f216d702700 1 -- xxx.xxx.58.15:0/758879303 --> [v2:xxx.xxx.58.2:6816/8556,v1:xxx.xxx.58.2:6817/8556] -- osd_op(unknown.0.0:238 18.651 18:8a75a7b2:::39078a70-7768-48c8-96a5-1e13ced83b5b.58017020.1_videos%2f7.mp4:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60419) v8 -- 0x7f21dc00a420 con 0x7f21dc007820 You can find the OSD part of the log here - https://pastebin.com/nGQw4ugd For the record the version of the cluster when I’m able to replicate this is ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Regards, Ondrej
On 20. 9. 2023, at 11:25, Ondřej Kukla <ondrej@kuuk.la> wrote:
I was checking the tracker again and I found already fixed issue that seems to be connected with this issue.
https://tracker.ceph.com/issues/44508
Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807
What I’m still not understanding is why this is only happening when using s3website api.
Is there someone who could shed some light on this?
Regards,
Ondrej _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 19. 9. 2023, at 10:49, Ondřej Kukla <ondrej@kuuk.la> wrote:
Hello,
In our deployment we are using the mix of s3 and s3website RGW. I’ve noticed strange behaviour when sending range requests to the s3website RGWs that I’m not able to replicate on the s3 ones.
I’ve created a simple wrk LUA script to test sending range requests on tiny ranges so the issue is easily seen.
When sending these requests against s3 RGW I can see that the amount of data read from Ceph is ± equivalent to what the RGW sends to the client. This change very dramatically when I’m doing the same test against s3website RGW. The read from Ceph is huge (3Gb/s compared to ~22Mb/s on s3 RGW) I seems to me like the RGW is reading the whole files and then sending just the range which is different then what s3 does.
I do not understand why would s3website need to read that much from Ceph and I believe this is a bug - I was looking through the tracker and wasn’t able to find anything related to s3website and range requests.
Did anyone else noticed this issue?
You can replicate it by running this wrk command wrk -t56 -c500 -d5m http://${rgwipaddress}:8080/${bucket}/videos/ -s wrk-range-small.lua
wrk script
-- Initialize the pseudo random number generator math.randomseed( os.time()) math.random(); math.random(); math.random()
i = 1
function request() if i == 8 then i = 1 end
local nrangefrom = math.random() local nrangeto = math.random(100) local path = wrk.path url = path..i..".mp4" wrk.headers["Range"] = nrangefrom.."-"..nrangeto i = i+1 return wrk.format(nil, url) end
Kind regards,
Ondrej _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io