ceph-mgr behind nginx reverse proxy
ceph-mgr is difficult to put behind an nginx reverse proxy configuration currently because of the HTTP return codes that it uses in different situations. We have 3 ceph-mgr servers (one on each of 3 monitors). Only 1 is ever active. We do not expose the monitors outside of the cluster, so we cannot point the browser directly at the manager IP address, instead we have an nginx proxy server setup on a separate administrative node in front of the cluster. When a reverse proxy asks an inactive manager for the root URL (/), the inactive mgr will return a 302 redirect code referencing the active mgr node - which is good, we can handle that. However, when the reverse proxy requests content from the inactive mgr, for example requesting something like "/runtime.26209474bfa8dc87a77c.js", the inactive mgr returns a 404 Not Found error instead of a 302 redirect. This makes it difficult (impossible?) to get all of the content if an inactive ceph-mgr is selected by the reverse proxy. It would be nice if any request to an inactive ceph-mgr returned a 302 with the correct location (and full URI path) to the client (the reverse proxy in this case). thanks, Wyllys Ingersoll
Hi Wyllys, when you're talking about "ceph-mgr" here, I assume you're referring to the Ceph Dashboard, correct? On 9/18/19 3:42 PM, Wyllys Ingersoll wrote:
ceph-mgr is difficult to put behind an nginx reverse proxy configuration currently because of the HTTP return codes that it uses in different situations.
We have 3 ceph-mgr servers (one on each of 3 monitors). Only 1 is ever active. We do not expose the monitors outside of the cluster, so we cannot point the browser directly at the manager IP address, instead we have an nginx proxy server setup on a separate administrative node in front of the cluster.
When a reverse proxy asks an inactive manager for the root URL (/), the inactive mgr will return a 302 redirect code referencing the active mgr node - which is good, we can handle that. However, when the reverse proxy requests content from the inactive mgr, for example requesting something like "/runtime.26209474bfa8dc87a77c.js", the inactive mgr returns a 404 Not Found error instead of a 302 redirect. This makes it difficult (impossible?) to get all of the content if an inactive ceph-mgr is selected by the reverse proxy.
It would be nice if any request to an inactive ceph-mgr returned a 302 with the correct location (and full URI path) to the client (the reverse proxy in this case).
You raise a very good point. Indeed, the current implementation makes it a bit challenging to put the dashboard behind a proxy. We have captured some of these concerns in the tracker here: https://tracker.ceph.com/issues/24662 Patch: https://github.com/ceph/ceph/pull/29088 This fix will also be backported into the Nautilus branch. However, you seem to be observing a somewhat different issue. I would have assumed that the redirection would apply to all objects and not just the top level URL - would you mind submitting a bug report about this for further investigation? Maybe this is related to https://tracker.ceph.com/issues/41843 by any chance? Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)
On 9/23/19 10:27 AM, Lenz Grimmer wrote:
However, you seem to be observing a somewhat different issue. I would have assumed that the redirection would apply to all objects and not just the top level URL - would you mind submitting a bug report about this for further investigation?
Never mind, Volker already submitted a tracker issue and submitted a pull request to fix this: https://tracker.ceph.com/issues/41930 https://github.com/ceph/ceph/pull/30478 If you like, you could apply this patch and check if this fixes the issue for you. Thanks, Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)
Yes, I was talking about dashboard. I ended up with a working nginx reverse proxy configuration that works well enough. Thanks for looking into it. Any chance these fixes get backported to Mimic ? -Wyllys On Mon, Sep 23, 2019 at 5:59 AM Lenz Grimmer <lgrimmer@suse.com> wrote:
On 9/23/19 10:27 AM, Lenz Grimmer wrote:
However, you seem to be observing a somewhat different issue. I would have assumed that the redirection would apply to all objects and not just the top level URL - would you mind submitting a bug report about this for further investigation?
Never mind, Volker already submitted a tracker issue and submitted a pull request to fix this:
https://tracker.ceph.com/issues/41930 https://github.com/ceph/ceph/pull/30478
If you like, you could apply this patch and check if this fixes the issue for you.
Thanks,
Lenz
-- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Yes, I was talking about dashboard.
I ended up with a working nginx reverse proxy configuration that works well enough. Thanks for looking into it.
Any chance these fixes get backported to Mimic ?
I don't think so. It will be backported to Nautilus only. Volker
-Wyllys
On Mon, Sep 23, 2019 at 5:59 AM Lenz Grimmer <lgrimmer@suse.com <mailto:lgrimmer@suse.com>> wrote:
On 9/23/19 10:27 AM, Lenz Grimmer wrote:
> However, you seem to be observing a somewhat different issue. I would > have assumed that the redirection would apply to all objects and not > just the top level URL - would you mind submitting a bug report about > this for further investigation?
Never mind, Volker already submitted a tracker issue and submitted a pull request to fix this:
https://tracker.ceph.com/issues/41930 https://github.com/ceph/ceph/pull/30478
If you like, you could apply this patch and check if this fixes the issue for you.
Thanks,
Lenz
-- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)
_______________________________________________ Dev mailing list -- dev@ceph.io <mailto:dev@ceph.io> To unsubscribe send an email to dev-leave@ceph.io <mailto:dev-leave@ceph.io>
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Volker Theile Software Engineer | Ceph | openATTIC SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany GF: Felix Imendörffer, HRB 247165 (AG München) Phone: +49 173 5876879 E-Mail: vtheile@suse.com
Hi Wyllys, On 9/23/19 4:30 PM, Wyllys Ingersoll wrote:
Yes, I was talking about dashboard.
I ended up with a working nginx reverse proxy configuration that works well enough. Thanks for looking into it.
OK, glad to hear you found a working solution.
Any chance these fixes get backported to Mimic ?
The patch should apply cleanly on Mimic, so I would not be against it. I've added it to the list of backport versions. Lenz -- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)
Great, thanks! On Fri, Sep 27, 2019 at 5:37 AM Lenz Grimmer <lgrimmer@suse.com> wrote:
Hi Wyllys,
On 9/23/19 4:30 PM, Wyllys Ingersoll wrote:
Yes, I was talking about dashboard.
I ended up with a working nginx reverse proxy configuration that works well enough. Thanks for looking into it.
OK, glad to hear you found a working solution.
Any chance these fixes get backported to Mimic ?
The patch should apply cleanly on Mimic, so I would not be against it. I've added it to the list of backport versions.
Lenz
-- SUSE Software Solutions Germany GmbH - Maxfeldstr. 5 - 90409 Nuernberg GF: Felix Imendörffer, HRB 247165 (AG Nürnberg)
Hi Lenz and Wyllys, issue https://tracker.ceph.com/issues/41930 has been created some days ago by myself to address this issue. The related PR is https://github.com/ceph/ceph/pull/30478. Regards Volker Am 23.09.19 um 10:27 schrieb Lenz Grimmer:
Hi Wyllys,
when you're talking about "ceph-mgr" here, I assume you're referring to the Ceph Dashboard, correct?
On 9/18/19 3:42 PM, Wyllys Ingersoll wrote:
ceph-mgr is difficult to put behind an nginx reverse proxy configuration currently because of the HTTP return codes that it uses in different situations.
We have 3 ceph-mgr servers (one on each of 3 monitors). Only 1 is ever active. We do not expose the monitors outside of the cluster, so we cannot point the browser directly at the manager IP address, instead we have an nginx proxy server setup on a separate administrative node in front of the cluster.
When a reverse proxy asks an inactive manager for the root URL (/), the inactive mgr will return a 302 redirect code referencing the active mgr node - which is good, we can handle that. However, when the reverse proxy requests content from the inactive mgr, for example requesting something like "/runtime.26209474bfa8dc87a77c.js", the inactive mgr returns a 404 Not Found error instead of a 302 redirect. This makes it difficult (impossible?) to get all of the content if an inactive ceph-mgr is selected by the reverse proxy.
It would be nice if any request to an inactive ceph-mgr returned a 302 with the correct location (and full URI path) to the client (the reverse proxy in this case). You raise a very good point. Indeed, the current implementation makes it a bit challenging to put the dashboard behind a proxy. We have captured some of these concerns in the tracker here:
https://tracker.ceph.com/issues/24662 Patch: https://github.com/ceph/ceph/pull/29088
This fix will also be backported into the Nautilus branch.
However, you seem to be observing a somewhat different issue. I would have assumed that the redirection would apply to all objects and not just the top level URL - would you mind submitting a bug report about this for further investigation?
Maybe this is related to https://tracker.ceph.com/issues/41843 by any chance?
Lenz
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Volker Theile Software Engineer | Ceph | openATTIC SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany GF: Felix Imendörffer, HRB 247165 (AG München) Phone: +49 173 5876879 E-Mail: vtheile@suse.com
I have done this using pacemaker just fine, when I access the VIP, it automatically goes to the active mon hosting dashboard. On Mon, Sep 23, 2019 at 3:58 PM Volker Theile <vtheile@suse.com> wrote:
Hi Lenz and Wyllys,
issue https://tracker.ceph.com/issues/41930 has been created some days ago by myself to address this issue. The related PR is https://github.com/ceph/ceph/pull/30478.
Regards Volker
Am 23.09.19 um 10:27 schrieb Lenz Grimmer:
Hi Wyllys,
when you're talking about "ceph-mgr" here, I assume you're referring to the Ceph Dashboard, correct?
On 9/18/19 3:42 PM, Wyllys Ingersoll wrote:
ceph-mgr is difficult to put behind an nginx reverse proxy configuration currently because of the HTTP return codes that it uses in different situations.
We have 3 ceph-mgr servers (one on each of 3 monitors). Only 1 is ever active. We do not expose the monitors outside of the cluster, so we cannot point the browser directly at the manager IP address, instead we have an nginx proxy server setup on a separate administrative node in front of the cluster.
When a reverse proxy asks an inactive manager for the root URL (/), the inactive mgr will return a 302 redirect code referencing the active mgr node - which is good, we can handle that. However, when the reverse proxy requests content from the inactive mgr, for example requesting something like "/runtime.26209474bfa8dc87a77c.js", the inactive mgr returns a 404 Not Found error instead of a 302 redirect. This makes it difficult (impossible?) to get all of the content if an inactive ceph-mgr is selected by the reverse proxy.
It would be nice if any request to an inactive ceph-mgr returned a 302 with the correct location (and full URI path) to the client (the reverse proxy in this case).
You raise a very good point. Indeed, the current implementation makes it a bit challenging to put the dashboard behind a proxy. We have captured some of these concerns in the tracker here: https://tracker.ceph.com/issues/24662 Patch: https://github.com/ceph/ceph/pull/29088
This fix will also be backported into the Nautilus branch.
However, you seem to be observing a somewhat different issue. I would have assumed that the redirection would apply to all objects and not just the top level URL - would you mind submitting a bug report about this for further investigation?
Maybe this is related to https://tracker.ceph.com/issues/41843 by any chance?
Lenz
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Volker Theile Software Engineer | Ceph | openATTIC SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nürnberg Germany GF: Felix Imendörffer, HRB 247165 (AG München) Phone: +49 173 5876879 E-Mail: vtheile@suse.com
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
participants (4)
-
Lenz Grimmer
-
solarflow99
-
Volker Theile
-
Wyllys Ingersoll