Hi Seena, The errno -5 is EIO, but as you saw it converts to 500--which is retryable (clients should re-try). As Yehuda said, the client could have stalled, but this error will also be generated if the client disconnects (e.g., timed out because RGW took too long). The logs should give a better indication what actually happened. Matt On Mon, Mar 23, 2020 at 9:00 AM Seena Fallah <seenafallah(a)gmail.com> wrote: -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

Hi again :) Is this normal? It fails with -5 err code but at the end request finish with 200! 2020-04-28 01:50:55.503 7f6982bf8700 0 iterate_obj() failed with -5 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj completing 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj op status=-5 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj http status=200 2020-04-28 01:50:55.503 7f6982bf8700 1 ====== req done req=0x7f6982bf18c0 op status=-5 http_status=200 latency=0.0320004s ====== On Mon, Mar 23, 2020 at 5:56 PM Seena Fallah <seenafallah(a)gmail.com> wrote: > > Thanks. I will increase it to see what happens on next time this issue occurred; > > On Mon, Mar 23, 2020 at 5:44 PM Matt Benjamin <mbenjami(a)redhat.com> wrote: >> >> Hi Seena, >> >> The errno -5 is EIO, but as you saw it converts to 500--which is >> retryable (clients should re-try). As Yehuda said, the client could >> have stalled, but this error will also be generated if the client >> disconnects (e.g., timed out because RGW took too long). The logs >> should give a better indication what actually happened. >> >> Matt >> >> On Mon, Mar 23, 2020 at 9:00 AM Seena Fallah <seenafallah(a)gmail.com> wrote: >> > >> > If it’s because of client why the status code changed to 500 (server error)? Isn’t it a client error? >> > >> > On Mon, Mar 23, 2020 at 5:26 PM Yehuda Sadeh-Weinraub <yehuda(a)redhat.com> wrote: >> >> >> >> On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah(a)gmail.com> wrote: >> >> > >> >> > Hi all. >> >> > I often get this error on my PUT requests. >> >> > >> >> > rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING: set_req_state_err err_no=5 resorting to 500 >> >> > rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ====== >> >> > >> >> > My put requests are always under 1s but this requests spend 1min! >> >> > Any logs I can read to find out what's going on with my cluster? >> >> >> >> Likely that rgw waits for more data from the client and eventually >> >> times out. This can happen due to various reasons. You can bump up rgw >> >> logs (debug rgw = 20), and try to look at what the client thinks it's >> >> sending vs what rgw receives. You can also try to look at tcp capture >> >> (via wireshark for example). >> >> >> >> Yehuda >> >> >> > _______________________________________________ >> > Dev mailing list -- dev(a)ceph.io >> > To unsubscribe send an email to dev-leave(a)ceph.io >> >> >> >> -- >> >> Matt Benjamin >> Red Hat, Inc. >> 315 West Huron Street, Suite 140A >> Ann Arbor, Michigan 48103 >> >> http://www.redhat.com/en/technologies/storage >> >> tel. 734-821-5101 >> fax. 734-769-8938 >> cel. 734-216-5309 >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io