Re: Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
Hello, Thank you very much to pickup the question and sorry for the late response. Yes, we are sending in cleartext also using HTTPS, but how it should be send if not like this? Also connected to this issue a bit, when we subscribe a bucket to a topic with non-ACL kafka topic, any operations (PUT or DELETE) is simply blocking and not returning. Not even any error response. $ s3cmd -c ~/.s3cfg put --add-header x-amz-meta-foo:bar3 certificate.pdf s3://vig-test WARNING: certificate.pdf: Owner groupname not known. Storing GID=1354917867 instead. WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: 'certificate.pdf' -> 's3://vig-test/certificate.pdf' [1 of 1] 65536 of 91224 71% in 0s 291.17 KB/s Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Yuval Lifshitz <ylifshit@redhat.com> Sent: Wednesday, April 21, 2021 10:34 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint Hi Istvan, Can you please share the relevant part for the radosgw log, indicating which input was invalid? The only way I managed to reproduce that error is by sending the request to a non-HTTPS radosgw (which does not seem to be your case). In such a case it replies with "InvalidInput" because we are trying to send user/password in cleartext. I used curl, similarly to what you did against a vstart cluster based off of master: https://paste.sh/SQ_8IrB5#BxBYbh1kTh15n7OKvjB5wEOM Yuval On Wed, Apr 21, 2021 at 11:23 AM Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: Hi Ceph Users, Here is the latest request I tried but still not working curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +0000' -H 'Authorization: AWS <accessid>:<signature>' -L -H 'content-type: application/x-www-form-urlencoded' -k -X POST https://servername -d Action=CreateTopic&Name=test-ceph-event-replication&Attributes.entry.8.key=push-endpoint&Attributes.entry.8.value=kafka://<username>:<password>@servername2:9093&Attributes.entry.5.key=use-ssl&Attributes.entry.5.value=true And the response I get is still Invalid Input <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx000000000000007993081-00607efbdd-1c7e96b-hkg</RequestId><HostId>1c7e96b-hkg-data</HostId></Error> Can someone please help with this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com><mailto:istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
On Tue, Apr 27, 2021 at 1:59 PM Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> wrote:
Hello,
Thank you very much to pickup the question and sorry for the late response.
Yes, we are sending in cleartext also using HTTPS, but how it should be send if not like this?
Also connected to this issue a bit, when we subscribe a bucket to a topic with non-ACL kafka topic, any operations (PUT or DELETE) is simply blocking and not returning. Not even any error response.
this would be the case when the kafka broker is down (or the parameters you provided to the topic were incorrect). a workaround for this issue is to mark the endpoint with "kafka-ack-level=none", this will not block for
if you send the user/password using HTTPS connection between the client and the RGW there should be no error. could you please provide the RGW debug log, to see why "invalid argument" was replied? the reply, but note that if the broker is down or misconfigured, the notification will be lost. a better option (if you are using "pacific" and up) is to mark the topic with the "persistent" flag. this would mean that even if the broker is down or misconfigured, the notification will be retired until successful, and, in addition, will not block the request.
$ s3cmd -c ~/.s3cfg put --add-header x-amz-meta-foo:bar3 certificate.pdf s3://vig-test WARNING: certificate.pdf: Owner groupname not known. Storing GID=1354917867 instead. WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: 'certificate.pdf' -> 's3://vig-test/certificate.pdf' [1 of 1] 65536 of 91224 71% in 0s 291.17 KB/s
Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
*From:* Yuval Lifshitz <ylifshit@redhat.com> *Sent:* Wednesday, April 21, 2021 10:34 PM *To:* Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> *Cc:* ceph-users@ceph.io *Subject:* Re: [ceph-users] Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
Hi Istvan,
Can you please share the relevant part for the radosgw log, indicating which input was invalid?
The only way I managed to reproduce that error is by sending the request to a non-HTTPS radosgw (which does not seem to be your case). In such a case it replies with "InvalidInput" because we are trying to send user/password in cleartext.
I used curl, similarly to what you did against a vstart cluster based off of master: https://paste.sh/SQ_8IrB5#BxBYbh1kTh15n7OKvjB5wEOM
Yuval
On Wed, Apr 21, 2021 at 11:23 AM Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> wrote:
Hi Ceph Users, Here is the latest request I tried but still not working
curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +0000' -H 'Authorization: AWS <accessid>:<signature>' -L -H 'content-type: application/x-www-form-urlencoded' -k -X POST https://servername -d Action=CreateTopic&Name=test-ceph-event-replication&Attributes.entry.8.key=push-endpoint&Attributes.entry.8.value=kafka://<username>:<password>@servername2:9093&Attributes.entry.5.key=use-ssl&Attributes.entry.5.value=true
And the response I get is still Invalid Input <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx000000000000007993081-00607efbdd-1c7e96b-hkg</RequestId><HostId>1c7e96b-hkg-data</HostId></Error> Can someone please help with this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> ---------------------------------------------------
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, What we have found seems like it is a blocking issue when I terminate https on a loadbalancer and between the loadbalancer and rgw http is the mode. So seems liket he ssl termination has to be done on the rgw and can't be done on the loadbalancer? Or how we can workaround it any idea? Here are the debug logs: With loadbalancer https endpoint: https://justpaste.it/5d93w Directly with rgw ip without loadbalancer: https://justpaste.it/9rn28 Both case the issue is like this: "endpoint validation error: sending password over insecure transport" To be honest I want to do the ssl on loadbalancer don't want to do on rgw. Maybe you can suggest something. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com --------------------------------------------------- -----Original Message----- From: Yuval Lifshitz <ylifshit@redhat.com> Sent: Tuesday, April 27, 2021 11:49 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io; Raveendran, Vigneshwaran (Agoda) <Vigneshwaran.Raveendran@agoda.com> Subject: [Suspicious newsletter] [ceph-users] Re: Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint On Tue, Apr 27, 2021 at 1:59 PM Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> wrote:
Hello,
Thank you very much to pickup the question and sorry for the late response.
Yes, we are sending in cleartext also using HTTPS, but how it should be send if not like this?
if you send the user/password using HTTPS connection between the client and the RGW there should be no error. could you please provide the RGW debug log, to see why "invalid argument" was replied?
Also connected to this issue a bit, when we subscribe a bucket to a topic with non-ACL kafka topic, any operations (PUT or DELETE) is simply blocking and not returning. Not even any error response.
this would be the case when the kafka broker is down (or the parameters you provided to the topic were incorrect). a workaround for this issue is to mark the endpoint with "kafka-ack-level=none", this will not block for the reply, but note that if the broker is down or misconfigured, the notification will be lost. a better option (if you are using "pacific" and up) is to mark the topic with the "persistent" flag. this would mean that even if the broker is down or misconfigured, the notification will be retired until successful, and, in addition, will not block the request.
$ s3cmd -c ~/.s3cfg put --add-header x-amz-meta-foo:bar3 certificate.pdf s3://vig-test WARNING: certificate.pdf: Owner groupname not known. Storing GID=1354917867 instead. WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: 'certificate.pdf' -> 's3://vig-test/certificate.pdf' [1 of 1] 65536 of 91224 71% in 0s 291.17 KB/s
Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
*From:* Yuval Lifshitz <ylifshit@redhat.com> *Sent:* Wednesday, April 21, 2021 10:34 PM *To:* Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> *Cc:* ceph-users@ceph.io *Subject:* Re: [ceph-users] Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
Hi Istvan,
Can you please share the relevant part for the radosgw log, indicating which input was invalid?
The only way I managed to reproduce that error is by sending the request to a non-HTTPS radosgw (which does not seem to be your case). In such a case it replies with "InvalidInput" because we are trying to send user/password in cleartext.
I used curl, similarly to what you did against a vstart cluster based off of master: https://paste.sh/SQ_8IrB5#BxBYbh1kTh15n7OKvjB5wEOM
Yuval
On Wed, Apr 21, 2021 at 11:23 AM Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> wrote:
Hi Ceph Users, Here is the latest request I tried but still not working
curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +0000' -H 'Authorization: AWS <accessid>:<signature>' -L -H 'content-type: application/x-www-form-urlencoded' -k -X POST https://servername -d Action=CreateTopic&Name=test-ceph-event-replication&Attributes.entry.8 .key=push-endpoint&Attributes.entry.8.value=kafka://<username>:<passwo rd>@servername2:9093&Attributes.entry.5.key=use-ssl&Attributes.entry.5 .value=true
And the response I get is still Invalid Input <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx0000000 00000007993081-00607efbdd-1c7e96b-hkg</RequestId><HostId>1c7e96b-hkg-d ata</HostId></Error> Can someone please help with this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> ---------------------------------------------------
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
I don't think there is a way around that. the RGW code does not allow user/password on non-ssl transport. what is the issue with SSL between the balancer and the RGW? if you have issues with self-signed certificates, maybe there is a way on the balancer to not verify them? On Wed, Apr 28, 2021 at 1:18 PM Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> wrote:
Hi,
What we have found seems like it is a blocking issue when I terminate https on a loadbalancer and between the loadbalancer and rgw http is the mode. So seems liket he ssl termination has to be done on the rgw and can't be done on the loadbalancer? Or how we can workaround it any idea?
Here are the debug logs:
With loadbalancer https endpoint: https://justpaste.it/5d93w Directly with rgw ip without loadbalancer: https://justpaste.it/9rn28
Both case the issue is like this: "endpoint validation error: sending password over insecure transport"
To be honest I want to do the ssl on loadbalancer don't want to do on rgw. Maybe you can suggest something.
Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
-----Original Message----- From: Yuval Lifshitz <ylifshit@redhat.com> Sent: Tuesday, April 27, 2021 11:49 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io; Raveendran, Vigneshwaran (Agoda) < Vigneshwaran.Raveendran@agoda.com> Subject: [Suspicious newsletter] [ceph-users] Re: Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
On Tue, Apr 27, 2021 at 1:59 PM Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> wrote:
Hello,
Thank you very much to pickup the question and sorry for the late response.
Yes, we are sending in cleartext also using HTTPS, but how it should be send if not like this?
if you send the user/password using HTTPS connection between the client and the RGW there should be no error. could you please provide the RGW debug log, to see why "invalid argument" was replied?
Also connected to this issue a bit, when we subscribe a bucket to a topic with non-ACL kafka topic, any operations (PUT or DELETE) is simply blocking and not returning. Not even any error response.
this would be the case when the kafka broker is down (or the parameters you provided to the topic were incorrect). a workaround for this issue is to mark the endpoint with "kafka-ack-level=none", this will not block for the reply, but note that if the broker is down or misconfigured, the notification will be lost. a better option (if you are using "pacific" and up) is to mark the topic with the "persistent" flag. this would mean that even if the broker is down or misconfigured, the notification will be retired until successful, and, in addition, will not block the request.
$ s3cmd -c ~/.s3cfg put --add-header x-amz-meta-foo:bar3 certificate.pdf s3://vig-test WARNING: certificate.pdf: Owner groupname not known. Storing GID=1354917867 instead. WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: 'certificate.pdf' -> 's3://vig-test/certificate.pdf' [1 of 1] 65536 of 91224 71% in 0s 291.17 KB/s
Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com ---------------------------------------------------
*From:* Yuval Lifshitz <ylifshit@redhat.com> *Sent:* Wednesday, April 21, 2021 10:34 PM *To:* Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> *Cc:* ceph-users@ceph.io *Subject:* Re: [ceph-users] Getting `InvalidInput` when trying to create a notification topic with Kafka endpoint
Hi Istvan,
Can you please share the relevant part for the radosgw log, indicating which input was invalid?
The only way I managed to reproduce that error is by sending the request to a non-HTTPS radosgw (which does not seem to be your case). In such a case it replies with "InvalidInput" because we are trying to send user/password in cleartext.
I used curl, similarly to what you did against a vstart cluster based off of master: https://paste.sh/SQ_8IrB5#BxBYbh1kTh15n7OKvjB5wEOM
Yuval
On Wed, Apr 21, 2021 at 11:23 AM Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com> wrote:
Hi Ceph Users, Here is the latest request I tried but still not working
curl -v -H 'Date: Tue, 20 Apr 2021 16:05:47 +0000' -H 'Authorization: AWS <accessid>:<signature>' -L -H 'content-type: application/x-www-form-urlencoded' -k -X POST https://servername -d Action=CreateTopic&Name=test-ceph-event-replication&Attributes.entry.8 .key=push-endpoint&Attributes.entry.8.value=kafka://<username>:<passwo rd>@servername2:9093&Attributes.entry.5.key=use-ssl&Attributes.entry.5 .value=true
And the response I get is still Invalid Input <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidInput</Code><RequestId>tx0000000 00000007993081-00607efbdd-1c7e96b-hkg</RequestId><HostId>1c7e96b-hkg-d ata</HostId></Error> Can someone please help with this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> ---------------------------------------------------
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
participants (2)
-
Szabo, Istvan (Agoda)
-
Yuval Lifshitz