Re: RGW access logs with bucket name
Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi, I am looking forward to move our logs from /var/log/ceph/ceph-client...log to our logaggregator.
Is there a way to have the bucket name in the log file?
Or can I write the rgw_enable_ops_log into a file? Maybe I could work with this.
Cheers and happy new year Boris
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
It has the full url begins with the bucket name in the beast logs http requests, hasn’t it? Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- On 2023. Mar 30., at 17:44, Boris Behrens <bb@kervyn.de> wrote: Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________ Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb@kervyn.de>: Hi, I am looking forward to move our logs from /var/log/ceph/ceph-client...log to our logaggregator. Is there a way to have the bucket name in the log file? Or can I write the rgw_enable_ops_log into a file? Maybe I could work with this. Cheers and happy new year Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Sadly not. I only see the the path/query of a request, but not the hostname. So when a bucket is accessed via hostname (https://bucket.TLD/object?query) I only see the object and the query (GET /object?query). When a bucket is accessed bia path (https://TLD/bucket/object?query) I can see also the bucket in the log (GET bucket/object?query) Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com>:
It has the full url begins with the bucket name in the beast logs http requests, hasn’t it?
Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
On 2023. Mar 30., at 17:44, Boris Behrens <bb@kervyn.de> wrote:
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Bringing up that topic again: is it possible to log the bucket name in the rgw client logs?
currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object.
Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi,
I am looking forward to move our logs from
/var/log/ceph/ceph-client...log to our logaggregator.
Is there a way to have the bucket name in the log file?
Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
this.
Cheers and happy new year
Boris
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
Hi Boris, I found that you need to use debug_rgw=10 to see the bucket name :-/ e.g. 2023-10-28T19:55:42.288+0000 7f34dde06700 10 req 3268931155513085118 0.000000000s s->object=... s->bucket=xyz-bucket-123 Did you find a more convenient way in the meantime? I think we should log bucket name at level 1. Cheers, Dan -- Dan van der Ster CTO Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanderster@clyso.com Try our Ceph Analyzer: https://analyzer.clyso.com On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens <bb@kervyn.de> wrote:
Sadly not. I only see the the path/query of a request, but not the hostname. So when a bucket is accessed via hostname (https://bucket.TLD/object?query) I only see the object and the query (GET /object?query). When a bucket is accessed bia path (https://TLD/bucket/object?query) I can see also the bucket in the log (GET bucket/object?query)
Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com>:
It has the full url begins with the bucket name in the beast logs http requests, hasn’t it?
Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
On 2023. Mar 30., at 17:44, Boris Behrens <bb@kervyn.de> wrote:
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Bringing up that topic again: is it possible to log the bucket name in the rgw client logs?
currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object.
Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb@kervyn.de>:
Hi,
I am looking forward to move our logs from
/var/log/ceph/ceph-client...log to our logaggregator.
Is there a way to have the bucket name in the log file?
Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
this.
Cheers and happy new year
Boris
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Dan, we are currently moving all the logging into lua scripts, so it is not an issue anymore for us. Thanks ps: the ceph analyzer is really cool. plusplus Am Sa., 28. Okt. 2023 um 22:03 Uhr schrieb Dan van der Ster < dan.vanderster@clyso.com>:
Hi Boris,
I found that you need to use debug_rgw=10 to see the bucket name :-/
e.g. 2023-10-28T19:55:42.288+0000 7f34dde06700 10 req 3268931155513085118 0.000000000s s->object=... s->bucket=xyz-bucket-123
Did you find a more convenient way in the meantime? I think we should log bucket name at level 1.
Cheers, Dan
-- Dan van der Ster CTO
Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanderster@clyso.com
Try our Ceph Analyzer: https://analyzer.clyso.com
On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens <bb@kervyn.de> wrote:
Sadly not. I only see the the path/query of a request, but not the hostname. So when a bucket is accessed via hostname (
https://bucket.TLD/object?query)
I only see the object and the query (GET /object?query). When a bucket is accessed bia path (https://TLD/bucket/object?query) I can see also the bucket in the log (GET bucket/object?query)
Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com>:
It has the full url begins with the bucket name in the beast logs http requests, hasn’t it?
Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
On 2023. Mar 30., at 17:44, Boris Behrens <bb@kervyn.de> wrote:
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Bringing up that topic again: is it possible to log the bucket name in the rgw client logs?
currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object.
Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb@kervyn.de :
Hi,
I am looking forward to move our logs from
/var/log/ceph/ceph-client...log to our logaggregator.
Is there a way to have the bucket name in the log file?
Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
this.
Cheers and happy new year
Boris
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.
another option is to enable the rgw ops log, which includes the bucket name for each request the http access log line that's visible at log level 1 follows a known apache format that users can scrape, so i've resisted adding extra s3-specific stuff like bucket/object names there. there was some recent discussion around this in https://github.com/ceph/ceph/pull/50350, which had originally extended that access log line On Mon, Oct 30, 2023 at 6:03 AM Boris Behrens <bb@kervyn.de> wrote:
Hi Dan,
we are currently moving all the logging into lua scripts, so it is not an issue anymore for us.
Thanks
ps: the ceph analyzer is really cool. plusplus
Am Sa., 28. Okt. 2023 um 22:03 Uhr schrieb Dan van der Ster < dan.vanderster@clyso.com>:
Hi Boris,
I found that you need to use debug_rgw=10 to see the bucket name :-/
e.g. 2023-10-28T19:55:42.288+0000 7f34dde06700 10 req 3268931155513085118 0.000000000s s->object=... s->bucket=xyz-bucket-123
Did you find a more convenient way in the meantime? I think we should log bucket name at level 1.
Cheers, Dan
-- Dan van der Ster CTO
Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanderster@clyso.com
Try our Ceph Analyzer: https://analyzer.clyso.com
On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens <bb@kervyn.de> wrote:
Sadly not. I only see the the path/query of a request, but not the hostname. So when a bucket is accessed via hostname (
https://bucket.TLD/object?query)
I only see the object and the query (GET /object?query). When a bucket is accessed bia path (https://TLD/bucket/object?query) I can see also the bucket in the log (GET bucket/object?query)
Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com>:
It has the full url begins with the bucket name in the beast logs http requests, hasn’t it?
Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
On 2023. Mar 30., at 17:44, Boris Behrens <bb@kervyn.de> wrote:
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Bringing up that topic again: is it possible to log the bucket name in the rgw client logs?
currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object.
Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb@kervyn.de :
Hi,
I am looking forward to move our logs from
/var/log/ceph/ceph-client...log to our logaggregator.
Is there a way to have the bucket name in the log file?
Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
this.
Cheers and happy new year
Boris
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Using the ops log is a good option -- I had missed that it can now log to a file. In Quincy: # ceph config set global rgw_ops_log_rados false # ceph config set global rgw_ops_log_file_path '/var/log/ceph/ops-log-$cluster-$name.log' # ceph config set global rgw_enable_ops_log true Then restart all RGWs. Thanks! Dan -- Dan van der Ster CTO Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanderster@clyso.com We are hiring: https://www.clyso.com/jobs/ On Mon, Oct 30, 2023 at 7:19 AM Casey Bodley <cbodley@redhat.com> wrote:
another option is to enable the rgw ops log, which includes the bucket name for each request
the http access log line that's visible at log level 1 follows a known apache format that users can scrape, so i've resisted adding extra s3-specific stuff like bucket/object names there. there was some recent discussion around this in https://github.com/ceph/ceph/pull/50350, which had originally extended that access log line
On Mon, Oct 30, 2023 at 6:03 AM Boris Behrens <bb@kervyn.de> wrote:
Hi Dan,
we are currently moving all the logging into lua scripts, so it is not an issue anymore for us.
Thanks
ps: the ceph analyzer is really cool. plusplus
Am Sa., 28. Okt. 2023 um 22:03 Uhr schrieb Dan van der Ster < dan.vanderster@clyso.com>:
Hi Boris,
I found that you need to use debug_rgw=10 to see the bucket name :-/
e.g. 2023-10-28T19:55:42.288+0000 7f34dde06700 10 req 3268931155513085118 0.000000000s s->object=... s->bucket=xyz-bucket-123
Did you find a more convenient way in the meantime? I think we should log bucket name at level 1.
Cheers, Dan
-- Dan van der Ster CTO
Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanderster@clyso.com
Try our Ceph Analyzer: https://analyzer.clyso.com
On Thu, Mar 30, 2023 at 4:15 AM Boris Behrens <bb@kervyn.de> wrote:
Sadly not. I only see the the path/query of a request, but not the hostname. So when a bucket is accessed via hostname (
https://bucket.TLD/object?query)
I only see the object and the query (GET /object?query). When a bucket is accessed bia path (https://TLD/bucket/object?query) I can see also the bucket in the log (GET bucket/object?query)
Am Do., 30. März 2023 um 12:58 Uhr schrieb Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com>:
It has the full url begins with the bucket name in the beast logs http requests, hasn’t it?
Istvan Szabo Staff Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
On 2023. Mar 30., at 17:44, Boris Behrens <bb@kervyn.de> wrote:
Email received from the internet. If in doubt, don't click any link nor open any attachment ! ________________________________
Bringing up that topic again: is it possible to log the bucket name in the rgw client logs?
currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object.
Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb@kervyn.de :
Hi,
I am looking forward to move our logs from
/var/log/ceph/ceph-client...log to our logaggregator.
Is there a way to have the bucket name in the log file?
Or can I write the rgw_enable_ops_log into a file? Maybe I could work with
this.
Cheers and happy new year
Boris
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
------------------------------ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Boris Behrens
-
Casey Bodley
-
Dan van der Ster
-
Szabo, Istvan (Agoda)