On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah(a)gmail.com> wrote: Likely that rgw waits for more data from the client and eventually times out. This can happen due to various reasons. You can bump up rgw logs (debug rgw = 20), and try to look at what the client thinks it's sending vs what rgw receives. You can also try to look at tcp capture (via wireshark for example). Yehuda

Hi Seena, The errno -5 is EIO, but as you saw it converts to 500--which is retryable (clients should re-try). As Yehuda said, the client could have stalled, but this error will also be generated if the client disconnects (e.g., timed out because RGW took too long). The logs should give a better indication what actually happened. Matt On Mon, Mar 23, 2020 at 9:00 AM Seena Fallah <seenafallah(a)gmail.com> wrote: -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

Hi again :) Is this normal? It fails with -5 err code but at the end request finish with 200! 2020-04-28 01:50:55.503 7f6982bf8700 0 iterate_obj() failed with -5 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj completing 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj op status=-5 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj http status=200 2020-04-28 01:50:55.503 7f6982bf8700 1 ====== req done req=0x7f6982bf18c0 op status=-5 http_status=200 latency=0.0320004s ====== On Mon, Mar 23, 2020 at 5:56 PM Seena Fallah <seenafallah(a)gmail.com> wrote: > > Thanks. I will increase it to see what happens on next time this issue occurred; > > On Mon, Mar 23, 2020 at 5:44 PM Matt Benjamin <mbenjami(a)redhat.com> wrote: >> >> Hi Seena, >> >> The errno -5 is EIO, but as you saw it converts to 500--which is >> retryable (clients should re-try). As Yehuda said, the client could >> have stalled, but this error will also be generated if the client >> disconnects (e.g., timed out because RGW took too long). The logs >> should give a better indication what actually happened. >> >> Matt >> >> On Mon, Mar 23, 2020 at 9:00 AM Seena Fallah <seenafallah(a)gmail.com> wrote: >> > >> > If it’s because of client why the status code changed to 500 (server error)? Isn’t it a client error? >> > >> > On Mon, Mar 23, 2020 at 5:26 PM Yehuda Sadeh-Weinraub <yehuda(a)redhat.com> wrote: >> >> >> >> On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah(a)gmail.com> wrote: >> >> > >> >> > Hi all. >> >> > I often get this error on my PUT requests. >> >> > >> >> > rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING: set_req_state_err err_no=5 resorting to 500 >> >> > rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ====== >> >> > >> >> > My put requests are always under 1s but this requests spend 1min! >> >> > Any logs I can read to find out what's going on with my cluster? >> >> >> >> Likely that rgw waits for more data from the client and eventually >> >> times out. This can happen due to various reasons. You can bump up rgw >> >> logs (debug rgw = 20), and try to look at what the client thinks it's >> >> sending vs what rgw receives. You can also try to look at tcp capture >> >> (via wireshark for example). >> >> >> >> Yehuda >> >> >> > _______________________________________________ >> > Dev mailing list -- dev(a)ceph.io >> > To unsubscribe send an email to dev-leave(a)ceph.io >> >> >> >> -- >> >> Matt Benjamin >> Red Hat, Inc. >> 315 West Huron Street, Suite 140A >> Ann Arbor, Michigan 48103 >> >> http://www.redhat.com/en/technologies/storage >> >> tel. 734-821-5101 >> fax. 734-769-8938 >> cel. 734-216-5309 >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io