Hi all. I often get this error on my PUT requests. rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING: set_req_state_err err_no=5 resorting to 500 rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ====== My put requests are always under 1s but this requests spend 1min! Any logs I can read to find out what's going on with my cluster?
On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah@gmail.com> wrote:
Hi all. I often get this error on my PUT requests.
rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING: set_req_state_err err_no=5 resorting to 500 rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ======
My put requests are always under 1s but this requests spend 1min! Any logs I can read to find out what's going on with my cluster?
Likely that rgw waits for more data from the client and eventually times out. This can happen due to various reasons. You can bump up rgw logs (debug rgw = 20), and try to look at what the client thinks it's sending vs what rgw receives. You can also try to look at tcp capture (via wireshark for example). Yehuda
If it’s because of client why the status code changed to 500 (server error)? Isn’t it a client error? On Mon, Mar 23, 2020 at 5:26 PM Yehuda Sadeh-Weinraub <yehuda@redhat.com> wrote:
On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah@gmail.com> wrote:
Hi all. I often get this error on my PUT requests.
rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING:
set_req_state_err err_no=5 resorting to 500
rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ======
My put requests are always under 1s but this requests spend 1min! Any logs I can read to find out what's going on with my cluster?
Likely that rgw waits for more data from the client and eventually times out. This can happen due to various reasons. You can bump up rgw logs (debug rgw = 20), and try to look at what the client thinks it's sending vs what rgw receives. You can also try to look at tcp capture (via wireshark for example).
Yehuda
Hi Seena, The errno -5 is EIO, but as you saw it converts to 500--which is retryable (clients should re-try). As Yehuda said, the client could have stalled, but this error will also be generated if the client disconnects (e.g., timed out because RGW took too long). The logs should give a better indication what actually happened. Matt On Mon, Mar 23, 2020 at 9:00 AM Seena Fallah <seenafallah@gmail.com> wrote:
If it’s because of client why the status code changed to 500 (server error)? Isn’t it a client error?
On Mon, Mar 23, 2020 at 5:26 PM Yehuda Sadeh-Weinraub <yehuda@redhat.com> wrote:
On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah@gmail.com> wrote:
Hi all. I often get this error on my PUT requests.
rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING: set_req_state_err err_no=5 resorting to 500 rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ======
My put requests are always under 1s but this requests spend 1min! Any logs I can read to find out what's going on with my cluster?
Likely that rgw waits for more data from the client and eventually times out. This can happen due to various reasons. You can bump up rgw logs (debug rgw = 20), and try to look at what the client thinks it's sending vs what rgw receives. You can also try to look at tcp capture (via wireshark for example).
Yehuda
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Thanks. I will increase it to see what happens on next time this issue occurred; On Mon, Mar 23, 2020 at 5:44 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Hi Seena,
The errno -5 is EIO, but as you saw it converts to 500--which is retryable (clients should re-try). As Yehuda said, the client could have stalled, but this error will also be generated if the client disconnects (e.g., timed out because RGW took too long). The logs should give a better indication what actually happened.
Matt
On Mon, Mar 23, 2020 at 9:00 AM Seena Fallah <seenafallah@gmail.com> wrote:
If it’s because of client why the status code changed to 500 (server
error)? Isn’t it a client error?
On Mon, Mar 23, 2020 at 5:26 PM Yehuda Sadeh-Weinraub <yehuda@redhat.com>
wrote:
On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah@gmail.com>
wrote:
Hi all. I often get this error on my PUT requests.
rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING:
set_req_state_err err_no=5 resorting to 500
rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ======
My put requests are always under 1s but this requests spend 1min! Any logs I can read to find out what's going on with my cluster?
Likely that rgw waits for more data from the client and eventually times out. This can happen due to various reasons. You can bump up rgw logs (debug rgw = 20), and try to look at what the client thinks it's sending vs what rgw receives. You can also try to look at tcp capture (via wireshark for example).
Yehuda
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Hi again :) Is this normal? It fails with -5 err code but at the end request finish with 200! 2020-04-28 01:50:55.503 7f6982bf8700 0 iterate_obj() failed with -5 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj completing 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj op status=-5 2020-04-28 01:50:55.503 7f6982bf8700 2 req 6456720 0.032s s3:get_obj http status=200 2020-04-28 01:50:55.503 7f6982bf8700 1 ====== req done req=0x7f6982bf18c0 op status=-5 http_status=200 latency=0.0320004s ====== On Mon, Mar 23, 2020 at 5:56 PM Seena Fallah <seenafallah@gmail.com> wrote:
Thanks. I will increase it to see what happens on next time this issue occurred;
On Mon, Mar 23, 2020 at 5:44 PM Matt Benjamin <mbenjami@redhat.com> wrote:
Hi Seena,
The errno -5 is EIO, but as you saw it converts to 500--which is retryable (clients should re-try). As Yehuda said, the client could have stalled, but this error will also be generated if the client disconnects (e.g., timed out because RGW took too long). The logs should give a better indication what actually happened.
Matt
On Mon, Mar 23, 2020 at 9:00 AM Seena Fallah <seenafallah@gmail.com> wrote:
If it’s because of client why the status code changed to 500 (server error)? Isn’t it a client error?
On Mon, Mar 23, 2020 at 5:26 PM Yehuda Sadeh-Weinraub <yehuda@redhat.com> wrote:
On Mon, Mar 23, 2020 at 2:28 PM Seena Fallah <seenafallah@gmail.com> wrote:
Hi all. I often get this error on my PUT requests.
rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING: set_req_state_err err_no=5 resorting to 500 rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ======
My put requests are always under 1s but this requests spend 1min! Any logs I can read to find out what's going on with my cluster?
Likely that rgw waits for more data from the client and eventually times out. This can happen due to various reasons. You can bump up rgw logs (debug rgw = 20), and try to look at what the client thinks it's sending vs what rgw receives. You can also try to look at tcp capture (via wireshark for example).
Yehuda
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (3)
-
Matt Benjamin
-
Seena Fallah
-
Yehuda Sadeh-Weinraub