Adding dev list. We haven't talked through much of this in any detail in the orchestrator calls yet aside from a vague discussion about what should/shouldn't be in scope. On Thu, 28 Nov 2019, Paul Cuzner wrote:
On Thu, Nov 28, 2019 at 2:37 AM Sage Weil <sweil@redhat.com> wrote:
On Wed, 27 Nov 2019, Paul Cuzner wrote:
Hi,
I've got a working gist for the add/remove of the monitoring solution. https://gist.github.com/pcuzner/ac542ce3fa9a4699bb9310b1fd5095d0
I'm out for the next couple of days, but will get a PR raised next week to get this started properly.
For some reason it won't let me comment on that gist.
- I don't think we should install anything on the host outside of the unit file and /var/lib/ceph/$fsid/$thing. I suggest $thing be 'prometheus', 'alertmanager', 'node-exporter', 'grafana'. We could combine all but node-exporter into a single 'monitoring' thing but i'm worried this obscures things too much when, for example, the user might have an external prometheus but still need alertmanager, and so on.
So all the configs should live in /var/lib/ceph/$fsid/$thing/prometheus.yml and so on, and then bound to the right /etc/whatever location by the container config.
I struggle with this one. Channelling my inner sysadmin: "I expect config settings to be in /etc and data to be in /var/lib - that's what FHS says and that's how other systems look that I have to manage, so why does Ceph have to do things differently?"
1- Because it's a containerized service. Things are in etc inside the container, not outside. Sprinkling these configs in /etc mixes containerized service configs with the *host*'s configs, which seems very untidy to me. 2. Putting it all in /var/lib/ceph/whatever means it's find and clean up.
I'm also not sure of the value of fsid in the dir names. I can see the value if a host has to support multiple ceph clusters - but outside dev is that something that the community or our customers actually want?
Most deployments won't need it, but it will avoid a whole range of problems when they do. Especially when it becomes trivial to bootstrap clusters, you also make it trivial to make multiple clusters overlap on the same host. And, like above, it keeps things tidy.
The gist downloads the separate containers we need in parallel - which I think is a good thing! reduces time
Sure... that's something we could do regardless of whether it's a separate script of part of ceph-daemon. Probably what we actually want is for the ssh 'host add' commadn to kick off some prestaging of containers in the background so that the first daemon deployment doesn't wait for a container download at all.
IMO, having monitoring-add deploy grafana/prom and alert manager together by default is the way to go. TBH, when I started this, I was putting them all in the same pod under podman for management and treat them as a single unit - but having to support 'legacy' docker put an end to that :)
If a user wishes to use a separate prometheus, that will normally have it's own alertmanager too. Which alertmanager a prometheus server is defined in the prometheus.yml. With external prometheus, rules, alerts and receiver definitions are going to be an exercise for the reader. We'll need to document the settings, but the admin will need to apply them - in this scenario, we could possibly generate sample files that the admin can pick up and apply? To my mind deployment of monitoring has two pathways; default - "monitoring add" yields prom/grafana/alertmanager containers deployed to machine external-prom - "monitoring add" just deploys grafana, and points it's default data source at the external prom url. We're also making an assumption here that the prometheus server is open and doesn't require auth (OCP's prometheus for example has auth enabled)
I think it makes sense to focus on the out-of-the-box opinionated easy scenario vs the DIY case, in general at least. But I have a few questions... - In the DIY case, does it makes sense to leave the node-exporter to the reader too? Or might it make sense for us to help deploy the node-exporter, but they run the external/existing prometheus instance? - Likewise, the alertmanager is going to have a bunch of ceph-specific alerts configured, right? Might they want their own prom but we deploy our alerts? (Is there any dependency in the dashboard on a particular set of alerts in prometheus?) I'm guessing you think no in both these cases...
- Let's teach ceph-daemon how to do this, so that you do 'ceph-daemon deploy --fsid ... --name prometheus.foo -i input.json'. ceph-daemon has the framework for opening firewall ports etc now... just add ports based on the daemon type.
TBH, I'd keep the monitoring containers away from the ceph daemons. They require different parameters, config files etc so why not keep them separate and keep the ceph logic clean. This also allows us to change monitoring without concerns over logic changes to normal ceph daemon management.
Okay, but mgr/ssh is still going to be wired up to deploy these. And to do so on a per-cluster, containerized basis... which means all of the infra in ceph-daemon will still be useful. It seems easiest to just add it there. Your points above seem to point toward simplifying the containers we deploy to just two containers, one that's one-per-cluster for prom+alertmanager+grafana, and one that's per-host for the node-exporter. But I think making it fit in nicely with the other ceph containers (e.g., /var/lib/ceph/$fsid/$thing) makes sense. Esp since we can just deploy these during bootstrap by default (unless some --external-prometheus is passed) and this all happens without the admin having to think about it.
WDYT?
I'm sure a lot of the above has already been discussed at length with the SuSE folks, so apologies for going over ground that you've already covered.
Not yet! :) sage
Adding dev list. We haven't talked through much of this in any detail in the orchestrator calls yet aside from a vague discussion about what should/shouldn't be in scope. Even if its an unpopular opinion: I think monitoring is absolutely out of scope for the orchestrator. Ceph works just fine without prometheus/grafana in place. Since I seem to be in the minority camp with this, I'd argue for at least making
On Wed, Nov 27, 2019 at 10:01:01PM +0000, Sage Weil wrote: this optional and integrating well with existing (further down called DIY case, though some might call what the orchestrator does DIY ;) monitoring deployments.
On Thu, 28 Nov 2019, Paul Cuzner wrote:
On Thu, Nov 28, 2019 at 2:37 AM Sage Weil <sweil@redhat.com> wrote:
On Wed, 27 Nov 2019, Paul Cuzner wrote:
Hi,
I've got a working gist for the add/remove of the monitoring solution. https://gist.github.com/pcuzner/ac542ce3fa9a4699bb9310b1fd5095d0
I'm out for the next couple of days, but will get a PR raised next week to get this started properly.
For some reason it won't let me comment on that gist.
- I don't think we should install anything on the host outside of the unit file and /var/lib/ceph/$fsid/$thing. I suggest $thing be 'prometheus', 'alertmanager', 'node-exporter', 'grafana'. We could combine all but node-exporter into a single 'monitoring' thing but i'm worried this obscures things too much when, for example, the user might have an external prometheus but still need alertmanager, and so on.
So all the configs should live in /var/lib/ceph/$fsid/$thing/prometheus.yml and so on, and then bound to the right /etc/whatever location by the container config.
I struggle with this one. Channelling my inner sysadmin: "I expect config settings to be in /etc and data to be in /var/lib - that's what FHS says and that's how other systems look that I have to manage, so why does Ceph have to do things differently?"
1- Because it's a containerized service. Things are in etc inside the container, not outside. Sprinkling these configs in /etc mixes containerized service configs with the *host*'s configs, which seems very untidy to me. 2. Putting it all in /var/lib/ceph/whatever means it's find and clean up.
I'm also not sure of the value of fsid in the dir names. I can see the value if a host has to support multiple ceph clusters - but outside dev is that something that the community or our customers actually want?
Most deployments won't need it, but it will avoid a whole range of problems when they do. Especially when it becomes trivial to bootstrap clusters, you also make it trivial to make multiple clusters overlap on the same host.
And, like above, it keeps things tidy.
The gist downloads the separate containers we need in parallel - which I think is a good thing! reduces time
Sure... that's something we could do regardless of whether it's a separate script of part of ceph-daemon. Probably what we actually want is for the ssh 'host add' commadn to kick off some prestaging of containers in the background so that the first daemon deployment doesn't wait for a container download at all.
IMO, having monitoring-add deploy grafana/prom and alert manager together by default is the way to go. TBH, when I started this, I was putting them all in the same pod under podman for management and treat them as a single unit - but having to support 'legacy' docker put an end to that :)
If a user wishes to use a separate prometheus, that will normally have it's own alertmanager too. Which alertmanager a prometheus server is defined in the prometheus.yml. With external prometheus, rules, alerts and receiver definitions are going to be an exercise for the reader. We'll need to document the settings, but the admin will need to apply them - in this scenario, we could possibly generate sample files that the admin can pick up and apply? To my mind deployment of monitoring has two pathways; default - "monitoring add" yields prom/grafana/alertmanager containers deployed to machine external-prom - "monitoring add" just deploys grafana, and points it's default data source at the external prom url. We're also making an assumption here that the prometheus server is open and doesn't require auth (OCP's prometheus for example has auth enabled)
I think it makes sense to focus on the out-of-the-box opinionated easy scenario vs the DIY case, in general at least. But I have a few questions...
I think this focus will leave some users in the dust. Monitoring with prometheus can get complex, especially if it is to be fault tolerant (which imho is important for confidence in such a system). Also typically users don't want several monitoring systems in their environment. So let's keep the case of existing prometheus systems in mind please.
- In the DIY case, does it makes sense to leave the node-exporter to the reader too? Or might it make sense for us to help deploy the node-exporter, but they run the external/existing prometheus instance?
- Likewise, the alertmanager is going to have a bunch of ceph-specific alerts configured, right? Might they want their own prom but we deploy our alerts? (Is there any dependency in the dashboard on a particular set of alerts in prometheus?)
I'm guessing you think no in both these cases...
What I'm missing from proposals I've seen so far is an interface to query the orchestrator for various prometheus bits. First and foremost the orchestrator should have a command that returns a prometheus file_sd_config of exporters that an external prometheus stack should scrape. Whether this is just the mgr exporter or also node_exporters (or others) depends on how far the orchestrator will take control. Alerts are currently handled as an rpm but could certainly be provided through a similar interface. At the very least, if the consensus will be that the orchestrator absolutely has to deploy everything itself, please at least provide an interface so that a federated setup is easily possible (an external prometheus scraping the orch-deployed prometheus) so that users don't have to care what the orchestrator does with monitoring (other then duplicating recorded metrics). See https://prometheus.io/docs/prometheus/latest/federation/#hierarchical-federa... I'd really like to encourage the orchestrator team to carefully think this through. Monitoring is (at least for some users) a critical infrastructure component with its own inherent complexity. I'm worried that just doing this in a best-effort fashion and not offering an alternative path if going to weaken the ceph ecosystem.
- Let's teach ceph-daemon how to do this, so that you do 'ceph-daemon deploy --fsid ... --name prometheus.foo -i input.json'. ceph-daemon has the framework for opening firewall ports etc now... just add ports based on the daemon type.
TBH, I'd keep the monitoring containers away from the ceph daemons. They require different parameters, config files etc so why not keep them separate and keep the ceph logic clean. This also allows us to change monitoring without concerns over logic changes to normal ceph daemon management.
Okay, but mgr/ssh is still going to be wired up to deploy these. And to do so on a per-cluster, containerized basis... which means all of the infra in ceph-daemon will still be useful. It seems easiest to just add it there.
Your points above seem to point toward simplifying the containers we deploy to just two containers, one that's one-per-cluster for prom+alertmanager+grafana, and one that's per-host for the node-exporter. But I think making it fit in nicely with the other ceph containers (e.g., /var/lib/ceph/$fsid/$thing) makes sense. Esp since we can just deploy these during bootstrap by default (unless some --external-prometheus is passed) and this all happens without the admin having to think about it.
WDYT?
I'm sure a lot of the above has already been discussed at length with the SuSE folks, so apologies for going over ground that you've already covered.
Not yet! :)
sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
On Thu, 28 Nov 2019, Jan Fajerski wrote:
Adding dev list. We haven't talked through much of this in any detail in the orchestrator calls yet aside from a vague discussion about what should/shouldn't be in scope. Even if its an unpopular opinion: I think monitoring is absolutely out of scope for the orchestrator. Ceph works just fine without prometheus/grafana in place. Since I seem to be in the minority camp with this, I'd argue for at least making
On Wed, Nov 27, 2019 at 10:01:01PM +0000, Sage Weil wrote: this optional and integrating well with existing (further down called DIY case, though some might call what the orchestrator does DIY ;) monitoring deployments.
Working with existing DIY metrics infra is the only non-negotiable requirement here. My goal is simply to have something that is simple and painless (and not necessarily even HA) just to make all the other dashboard graphs light up.
On Thu, 28 Nov 2019, Paul Cuzner wrote:
On Thu, Nov 28, 2019 at 2:37 AM Sage Weil <sweil@redhat.com> wrote:
On Wed, 27 Nov 2019, Paul Cuzner wrote:
Hi,
I've got a working gist for the add/remove of the monitoring solution. https://gist.github.com/pcuzner/ac542ce3fa9a4699bb9310b1fd5095d0
I'm out for the next couple of days, but will get a PR raised next week to get this started properly.
For some reason it won't let me comment on that gist.
- I don't think we should install anything on the host outside of the unit file and /var/lib/ceph/$fsid/$thing. I suggest $thing be 'prometheus', 'alertmanager', 'node-exporter', 'grafana'. We could combine all but node-exporter into a single 'monitoring' thing but i'm worried this obscures things too much when, for example, the user might have an external prometheus but still need alertmanager, and so on.
So all the configs should live in /var/lib/ceph/$fsid/$thing/prometheus.yml and so on, and then bound to the right /etc/whatever location by the container config.
I struggle with this one. Channelling my inner sysadmin: "I expect config settings to be in /etc and data to be in /var/lib - that's what FHS says and that's how other systems look that I have to manage, so why does Ceph have to do things differently?"
1- Because it's a containerized service. Things are in etc inside the container, not outside. Sprinkling these configs in /etc mixes containerized service configs with the *host*'s configs, which seems very untidy to me. 2. Putting it all in /var/lib/ceph/whatever means it's find and clean up.
I'm also not sure of the value of fsid in the dir names. I can see the value if a host has to support multiple ceph clusters - but outside dev is that something that the community or our customers actually want?
Most deployments won't need it, but it will avoid a whole range of problems when they do. Especially when it becomes trivial to bootstrap clusters, you also make it trivial to make multiple clusters overlap on the same host.
And, like above, it keeps things tidy.
The gist downloads the separate containers we need in parallel - which I think is a good thing! reduces time
Sure... that's something we could do regardless of whether it's a separate script of part of ceph-daemon. Probably what we actually want is for the ssh 'host add' commadn to kick off some prestaging of containers in the background so that the first daemon deployment doesn't wait for a container download at all.
IMO, having monitoring-add deploy grafana/prom and alert manager together by default is the way to go. TBH, when I started this, I was putting them all in the same pod under podman for management and treat them as a single unit - but having to support 'legacy' docker put an end to that :)
If a user wishes to use a separate prometheus, that will normally have it's own alertmanager too. Which alertmanager a prometheus server is defined in the prometheus.yml. With external prometheus, rules, alerts and receiver definitions are going to be an exercise for the reader. We'll need to document the settings, but the admin will need to apply them - in this scenario, we could possibly generate sample files that the admin can pick up and apply? To my mind deployment of monitoring has two pathways; default - "monitoring add" yields prom/grafana/alertmanager containers deployed to machine external-prom - "monitoring add" just deploys grafana, and points it's default data source at the external prom url. We're also making an assumption here that the prometheus server is open and doesn't require auth (OCP's prometheus for example has auth enabled)
I think it makes sense to focus on the out-of-the-box opinionated easy scenario vs the DIY case, in general at least. But I have a few questions... I think this focus will leave some users in the dust. Monitoring with prometheus can get complex, especially if it is to be fault tolerant (which imho is important for confidence in such a system). Also typically users don't want several monitoring systems in their environment. So let's keep the case of existing prometheus systems in mind please.
That's what I want meant by 'vs' above... perhaps I should have said 'or'. Either we deploy something simple and opinionated, or the user attaches to their existing or self-configured setup. We don't probably need to worry about the various points in the middle ground where we manage only part of the metrics solution. (Also, I'm trying to use 'metrics' to mean prometheus etc, vs 'monitoring' which in my mind is nagios or pagerduty or whatever and presumably has a level of HA required, and/or needs to be external instead of baked-in.) sage
- In the DIY case, does it makes sense to leave the node-exporter to the reader too? Or might it make sense for us to help deploy the node-exporter, but they run the external/existing prometheus instance?
- Likewise, the alertmanager is going to have a bunch of ceph-specific alerts configured, right? Might they want their own prom but we deploy our alerts? (Is there any dependency in the dashboard on a particular set of alerts in prometheus?)
I'm guessing you think no in both these cases...
What I'm missing from proposals I've seen so far is an interface to query the orchestrator for various prometheus bits. First and foremost the orchestrator should have a command that returns a prometheus file_sd_config of exporters that an external prometheus stack should scrape. Whether this is just the mgr exporter or also node_exporters (or others) depends on how far the orchestrator will take control. Alerts are currently handled as an rpm but could certainly be provided through a similar interface.
At the very least, if the consensus will be that the orchestrator absolutely has to deploy everything itself, please at least provide an interface so that a federated setup is easily possible (an external prometheus scraping the orch-deployed prometheus) so that users don't have to care what the orchestrator does with monitoring (other then duplicating recorded metrics). See https://prometheus.io/docs/prometheus/latest/federation/#hierarchical-federa...
I'd really like to encourage the orchestrator team to carefully think this through. Monitoring is (at least for some users) a critical infrastructure component with its own inherent complexity. I'm worried that just doing this in a best-effort fashion and not offering an alternative path if going to weaken the ceph ecosystem.
- Let's teach ceph-daemon how to do this, so that you do 'ceph-daemon deploy --fsid ... --name prometheus.foo -i input.json'. ceph-daemon has the framework for opening firewall ports etc now... just add ports based on the daemon type.
TBH, I'd keep the monitoring containers away from the ceph daemons. They require different parameters, config files etc so why not keep them separate and keep the ceph logic clean. This also allows us to change monitoring without concerns over logic changes to normal ceph daemon management.
Okay, but mgr/ssh is still going to be wired up to deploy these. And to do so on a per-cluster, containerized basis... which means all of the infra in ceph-daemon will still be useful. It seems easiest to just add it there.
Your points above seem to point toward simplifying the containers we deploy to just two containers, one that's one-per-cluster for prom+alertmanager+grafana, and one that's per-host for the node-exporter. But I think making it fit in nicely with the other ceph containers (e.g., /var/lib/ceph/$fsid/$thing) makes sense. Esp since we can just deploy these during bootstrap by default (unless some --external-prometheus is passed) and this all happens without the admin having to think about it.
WDYT?
I'm sure a lot of the above has already been discussed at length with the SuSE folks, so apologies for going over ground that you've already covered.
Not yet! :)
sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Thu, Nov 28, 2019 at 02:26:36PM +0000, Sage Weil wrote: --snip--
I think it makes sense to focus on the out-of-the-box opinionated easy scenario vs the DIY case, in general at least. But I have a few questions... I think this focus will leave some users in the dust. Monitoring with prometheus can get complex, especially if it is to be fault tolerant (which imho is important for confidence in such a system). Also typically users don't want several monitoring systems in their environment. So let's keep the case of existing prometheus systems in mind please.
That's what I want meant by 'vs' above... perhaps I should have said 'or'. Either we deploy something simple and opinionated, or the user attaches to their existing or self-configured setup. We don't probably need to worry about the various points in the middle ground where we manage only part of the metrics solution.
I'm not sure we'll get off this easy. At the very least the prometheus mgr module is deployed by us. There is also an argument to be made for monitoring the things that we take control over, i.e. the containers we deploy (one node_exporter per container is a common setup) and maybe even the hosts that the orchestrator provisions.
(Also, I'm trying to use 'metrics' to mean prometheus etc, vs 'monitoring' which in my mind is nagios or pagerduty or whatever and presumably has a level of HA required, and/or needs to be external instead of baked-in.)
Not sure I understand that distinction. You mean metrics for the prometheus setup the orchestrator intents to install? (prometheus can certainly be a fully fledged monitoring stack). Jan
sage
- In the DIY case, does it makes sense to leave the node-exporter to the reader too? Or might it make sense for us to help deploy the node-exporter, but they run the external/existing prometheus instance?
- Likewise, the alertmanager is going to have a bunch of ceph-specific alerts configured, right? Might they want their own prom but we deploy our alerts? (Is there any dependency in the dashboard on a particular set of alerts in prometheus?)
I'm guessing you think no in both these cases...
What I'm missing from proposals I've seen so far is an interface to query the orchestrator for various prometheus bits. First and foremost the orchestrator should have a command that returns a prometheus file_sd_config of exporters that an external prometheus stack should scrape. Whether this is just the mgr exporter or also node_exporters (or others) depends on how far the orchestrator will take control. Alerts are currently handled as an rpm but could certainly be provided through a similar interface.
At the very least, if the consensus will be that the orchestrator absolutely has to deploy everything itself, please at least provide an interface so that a federated setup is easily possible (an external prometheus scraping the orch-deployed prometheus) so that users don't have to care what the orchestrator does with monitoring (other then duplicating recorded metrics). See https://prometheus.io/docs/prometheus/latest/federation/#hierarchical-federa...
I'd really like to encourage the orchestrator team to carefully think this through. Monitoring is (at least for some users) a critical infrastructure component with its own inherent complexity. I'm worried that just doing this in a best-effort fashion and not offering an alternative path if going to weaken the ceph ecosystem.
- Let's teach ceph-daemon how to do this, so that you do 'ceph-daemon deploy --fsid ... --name prometheus.foo -i input.json'. ceph-daemon has the framework for opening firewall ports etc now... just add ports based on the daemon type.
TBH, I'd keep the monitoring containers away from the ceph daemons. They require different parameters, config files etc so why not keep them separate and keep the ceph logic clean. This also allows us to change monitoring without concerns over logic changes to normal ceph daemon management.
Okay, but mgr/ssh is still going to be wired up to deploy these. And to do so on a per-cluster, containerized basis... which means all of the infra in ceph-daemon will still be useful. It seems easiest to just add it there.
Your points above seem to point toward simplifying the containers we deploy to just two containers, one that's one-per-cluster for prom+alertmanager+grafana, and one that's per-host for the node-exporter. But I think making it fit in nicely with the other ceph containers (e.g., /var/lib/ceph/$fsid/$thing) makes sense. Esp since we can just deploy these during bootstrap by default (unless some --external-prometheus is passed) and this all happens without the admin having to think about it.
WDYT?
I'm sure a lot of the above has already been discussed at length with the SuSE folks, so apologies for going over ground that you've already covered.
Not yet! :)
sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
Interesting discussion - but I don't want to lose sight of the original questions ceph-deamon make several deployment decisions at the moment that differs from existing deployment patterns. This is the first point that I wanted to raise. - it assumes that from Octopus onwards, the only deployment pattern we provide is container only. - it places all of Ceph's files (config and data) within /var/lib. In the past even with containers, we've still used /etc for config to align with FHS and, since the OS is package based, config from other packages adheries to FHS anyway - which makes Ceph different. - it uses fsid in path names and container names, just in case users want to run multiple ceph clusters on the same machine. IMO this adds complication to 100% of deployments, that may benefit 5% of the user base (numbers plucked out of the air on that one!) Perhaps, all of these design points trace back to a single idea - support multiple ceph clusters on the same set of machine(s). Is this the goal? Is this want Ceph users want? Now picking up on the scope issue for the orchestrator - apologies if this sounds like a manifesto...I'm a "usability" addict! IMO, our collective goal should be to drive ease of use and Ceph adoption beyond Linux geeks. If that's a view that resonates, I think the orchestrator has critical role to play to enable that strategy Personally I'd would like to see the orchestrator evolve over time to become the automation engine that enables an open source ecosystem around Ceph; - provide a default implementation for monitoring/alerting/metrics - this can be simple and doesn't need HA - as Sage has already mentioned - samba/ganesha deployment, loud balancers to improve radosgw etc etc - integration with platform management (why not show in the ceph dashboard whether you have patches outstanding against your host, or the host has a bad PSU) - enable the sysadmin to work more efficiently on Ceph, and maybe they'd prefer it over other platforms. We absolutely still need to support DIY configurations - but having a strategy that delivers a better out-of-the-box Ceph experience is surely our goal. </soapbox> On Fri, Nov 29, 2019 at 8:22 PM Jan Fajerski <jfajerski@suse.com> wrote:
I think it makes sense to focus on the out-of-the-box opinionated easy scenario vs the DIY case, in general at least. But I have a few questions... I think this focus will leave some users in the dust. Monitoring with
On Thu, Nov 28, 2019 at 02:26:36PM +0000, Sage Weil wrote: --snip-- prometheus
can get complex, especially if it is to be fault tolerant (which imho is important for confidence in such a system). Also typically users don't want several monitoring systems in their environment. So let's keep the case of existing prometheus systems in mind please.
That's what I want meant by 'vs' above... perhaps I should have said 'or'. Either we deploy something simple and opinionated, or the user attaches to their existing or self-configured setup. We don't probably need to worry about the various points in the middle ground where we manage only part of the metrics solution.
I'm not sure we'll get off this easy. At the very least the prometheus mgr module is deployed by us. There is also an argument to be made for monitoring the things that we take control over, i.e. the containers we deploy (one node_exporter per container is a common setup) and maybe even the hosts that the orchestrator provisions.
(Also, I'm trying to use 'metrics' to mean prometheus etc, vs 'monitoring' which in my mind is nagios or pagerduty or whatever and presumably has a level of HA required, and/or needs to be external instead of baked-in.)
Not sure I understand that distinction. You mean metrics for the prometheus setup the orchestrator intents to install? (prometheus can certainly be a fully fledged monitoring stack).
Jan
sage
- In the DIY case, does it makes sense to leave the node-exporter to
reader too? Or might it make sense for us to help deploy the node-exporter, but they run the external/existing prometheus instance?
- Likewise, the alertmanager is going to have a bunch of ceph-specific alerts configured, right? Might they want their own prom but we deploy our alerts? (Is there any dependency in the dashboard on a particular set of alerts in prometheus?)
I'm guessing you think no in both these cases...
What I'm missing from proposals I've seen so far is an interface to query the orchestrator for various prometheus bits. First and foremost the orchestrator should have a command that returns a prometheus file_sd_config of exporters that an external prometheus stack should scrape. Whether this is just the mgr exporter or also node_exporters (or others) depends on how far the orchestrator will take control. Alerts are currently handled as an rpm but could certainly be provided
similar interface.
At the very least, if the consensus will be that the orchestrator absolutely has to deploy everything itself, please at least provide an interface so
federated setup is easily possible (an external prometheus scraping the orch-deployed prometheus) so that users don't have to care what the orchestrator does with monitoring (other then duplicating recorded metrics). See
https://prometheus.io/docs/prometheus/latest/federation/#hierarchical-federa...
I'd really like to encourage the orchestrator team to carefully think
through. Monitoring is (at least for some users) a critical infrastructure component with its own inherent complexity. I'm worried that just doing
a best-effort fashion and not offering an alternative path if going to weaken the ceph ecosystem.
- Let's teach ceph-daemon how to do this, so that you do
'ceph-daemon
deploy --fsid ... --name prometheus.foo -i input.json'. ceph-daemon has the framework for opening firewall ports etc now... just add
the through a that a this this in ports
based on the daemon type.
TBH, I'd keep the monitoring containers away from the ceph daemons. They require different parameters, config files etc so why not keep them separate and keep the ceph logic clean. This also allows us to change monitoring without concerns over logic changes to normal ceph daemon management.
Okay, but mgr/ssh is still going to be wired up to deploy these. And to do so on a per-cluster, containerized basis... which means all of the infra in ceph-daemon will still be useful. It seems easiest to just add it there.
Your points above seem to point toward simplifying the containers we deploy to just two containers, one that's one-per-cluster for prom+alertmanager+grafana, and one that's per-host for the node-exporter. But I think making it fit in nicely with the other ceph containers (e.g., /var/lib/ceph/$fsid/$thing) makes sense. Esp since we can just deploy these during bootstrap by default (unless some --external-prometheus is passed) and this all happens without the admin having to think about it.
WDYT?
I'm sure a lot of the above has already been discussed at length with the SuSE folks, so apologies for going over ground that you've already covered.
Not yet! :)
sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Did you mean ceph-daemon, or perhaps ceph-volume?
Perhaps, all of these design points trace back to a single idea - support multiple ceph clusters on the same set of machine(s). Is this the goal? Is this want Ceph users want?
Custom cluster names were originally intended for this purpose, though there were some gaps in implemention. When custom cluster names began to be deprecated, it was claimed that almost nobody ever made us of this functionality. So unless there’s been a radical shift, multiple clusters on the same set of systems is not a concern going forward.
On Wed, Dec 4, 2019 at 11:27 AM Anthony D'Atri <aad@dreamsnake.net> wrote:
Did you mean ceph-daemon, or perhaps ceph-volume?
ceph-daemon - https://github.com/ceph/ceph/blob/master/src/ceph-daemon/ceph-daemon the new installation/management feature for Octopus - https://docs.ceph.com/docs/master/bootstrap/
Perhaps, all of these design points trace back to a single idea - support multiple ceph clusters on the same set of machine(s). Is this the goal? Is this want Ceph users want?
Custom cluster names were originally intended for this purpose, though there were some gaps in implemention. When custom cluster names began to be deprecated, it was claimed that almost nobody ever made us of this functionality. So unless there’s been a radical shift, multiple clusters on the same set of systems is not a concern going forward.
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Wed, 4 Dec 2019, Paul Cuzner wrote:
Interesting discussion - but I don't want to lose sight of the original questions
ceph-deamon make several deployment decisions at the moment that differs from existing deployment patterns. This is the first point that I wanted to raise. - it assumes that from Octopus onwards, the only deployment pattern we provide is container only. - it places all of Ceph's files (config and data) within /var/lib. In the past even with containers, we've still used /etc for config to align with FHS and, since the OS is package based, config from other packages adheries to FHS anyway - which makes Ceph different. - it uses fsid in path names and container names, just in case users want to run multiple ceph clusters on the same machine. IMO this adds complication to 100% of deployments, that may benefit 5% of the user base (numbers plucked out of the air on that one!)
Perhaps, all of these design points trace back to a single idea - support multiple ceph clusters on the same set of machine(s). Is this the goal? Is this want Ceph users want?
It was one of my goals. A few reasons: - It's easy and clean. - These users do exist. - When we deprecated this behaviour before, our justification was "you should be using containers". Well, here we are. - Rook allows this (with the (current) caveat that you can't put mons from multiple clusters on the same host/IP if they're using default ports). - The paths for rook are also convoluted like this, nested under the kubernetes namespace name. - The pain of weird paths is mitigated when you use enter a containers (or use the shell container). - tab-completion works both for path names and systemd service names. Also, ceph-daemon shell and similar commands will figure out the fsid themselves when there is a single cluster on the host.
Now picking up on the scope issue for the orchestrator - apologies if this sounds like a manifesto...I'm a "usability" addict!
IMO, our collective goal should be to drive ease of use and Ceph adoption beyond Linux geeks. If that's a view that resonates, I think the orchestrator has critical role to play to enable that strategy Personally I'd would like to see the orchestrator evolve over time to become the automation engine that enables an open source ecosystem around Ceph; - provide a default implementation for monitoring/alerting/metrics - this can be simple and doesn't need HA - as Sage has already mentioned - samba/ganesha deployment, loud balancers to improve radosgw etc etc - integration with platform management (why not show in the ceph dashboard whether you have patches outstanding against your host, or the host has a bad PSU) - enable the sysadmin to work more efficiently on Ceph, and maybe they'd prefer it over other platforms.
We absolutely still need to support DIY configurations - but having a strategy that delivers a better out-of-the-box Ceph experience is surely our goal.
</soapbox>
+1 sage
On Fri, Nov 29, 2019 at 8:22 PM Jan Fajerski <jfajerski@suse.com> wrote:
I think it makes sense to focus on the out-of-the-box opinionated easy scenario vs the DIY case, in general at least. But I have a few questions... I think this focus will leave some users in the dust. Monitoring with
On Thu, Nov 28, 2019 at 02:26:36PM +0000, Sage Weil wrote: --snip-- prometheus
can get complex, especially if it is to be fault tolerant (which imho is important for confidence in such a system). Also typically users don't want several monitoring systems in their environment. So let's keep the case of existing prometheus systems in mind please.
That's what I want meant by 'vs' above... perhaps I should have said 'or'. Either we deploy something simple and opinionated, or the user attaches to their existing or self-configured setup. We don't probably need to worry about the various points in the middle ground where we manage only part of the metrics solution.
I'm not sure we'll get off this easy. At the very least the prometheus mgr module is deployed by us. There is also an argument to be made for monitoring the things that we take control over, i.e. the containers we deploy (one node_exporter per container is a common setup) and maybe even the hosts that the orchestrator provisions.
(Also, I'm trying to use 'metrics' to mean prometheus etc, vs 'monitoring' which in my mind is nagios or pagerduty or whatever and presumably has a level of HA required, and/or needs to be external instead of baked-in.)
Not sure I understand that distinction. You mean metrics for the prometheus setup the orchestrator intents to install? (prometheus can certainly be a fully fledged monitoring stack).
Jan
sage
- In the DIY case, does it makes sense to leave the node-exporter to
reader too? Or might it make sense for us to help deploy the node-exporter, but they run the external/existing prometheus instance?
- Likewise, the alertmanager is going to have a bunch of ceph-specific alerts configured, right? Might they want their own prom but we deploy our alerts? (Is there any dependency in the dashboard on a particular set of alerts in prometheus?)
I'm guessing you think no in both these cases...
What I'm missing from proposals I've seen so far is an interface to query the orchestrator for various prometheus bits. First and foremost the orchestrator should have a command that returns a prometheus file_sd_config of exporters that an external prometheus stack should scrape. Whether this is just the mgr exporter or also node_exporters (or others) depends on how far the orchestrator will take control. Alerts are currently handled as an rpm but could certainly be provided
similar interface.
At the very least, if the consensus will be that the orchestrator absolutely has to deploy everything itself, please at least provide an interface so
federated setup is easily possible (an external prometheus scraping the orch-deployed prometheus) so that users don't have to care what the orchestrator does with monitoring (other then duplicating recorded metrics). See
https://prometheus.io/docs/prometheus/latest/federation/#hierarchical-federa...
I'd really like to encourage the orchestrator team to carefully think
through. Monitoring is (at least for some users) a critical infrastructure component with its own inherent complexity. I'm worried that just doing
a best-effort fashion and not offering an alternative path if going to weaken the ceph ecosystem.
> - Let's teach ceph-daemon how to do this, so that you do
'ceph-daemon
> deploy --fsid ... --name prometheus.foo -i input.json'. ceph-daemon > has the framework for opening firewall ports etc now... just add
the through a that a this this in ports
> based on the daemon type. >
TBH, I'd keep the monitoring containers away from the ceph daemons. They require different parameters, config files etc so why not keep them separate and keep the ceph logic clean. This also allows us to change monitoring without concerns over logic changes to normal ceph daemon management.
Okay, but mgr/ssh is still going to be wired up to deploy these. And to do so on a per-cluster, containerized basis... which means all of the infra in ceph-daemon will still be useful. It seems easiest to just add it there.
Your points above seem to point toward simplifying the containers we deploy to just two containers, one that's one-per-cluster for prom+alertmanager+grafana, and one that's per-host for the node-exporter. But I think making it fit in nicely with the other ceph containers (e.g., /var/lib/ceph/$fsid/$thing) makes sense. Esp since we can just deploy these during bootstrap by default (unless some --external-prometheus is passed) and this all happens without the admin having to think about it.
> WDYT? > > I'm sure a lot of the above has already been discussed at length with the SuSE folks, so apologies for going over ground that you've already covered.
Not yet! :)
sage _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 2019-12-04T00:12:27, Sage Weil <sage@newdream.net> wrote:
Perhaps, all of these design points trace back to a single idea - support multiple ceph clusters on the same set of machine(s). Is this the goal? Is this want Ceph users want? It was one of my goals. A few reasons:
So I agree this is useful. Glad to have this on the map again. I'm not sure how large this percentage is - the environments that I am aware of that'd require segregation into multiple clusters would also frown on using the same CPU/OS instance for running them, split out disks wouldn't be enough -, but I see it could be helpful. On the soapbox, though:
Now picking up on the scope issue for the orchestrator - apologies if this sounds like a manifesto...I'm a "usability" addict!
IMO, our collective goal should be to drive ease of use and Ceph adoption beyond Linux geeks. If that's a view that resonates, I think the orchestrator has critical role to play to enable that strategy Personally I'd would like to see the orchestrator evolve over time to become the automation engine that enables an open source ecosystem around Ceph; - provide a default implementation for monitoring/alerting/metrics - this can be simple and doesn't need HA - as Sage has already mentioned
Doesn't it though - not being alerted when your cluster enters a degraded state is pretty dangerous from an operational point of view. And as soon as users start relying on those metrics, losing them is also not an option. We can't not deploy monitoring/alerting/metrics, since they're essential to operating a Ceph cluster. But I think we're deluding ourselves if we even pretend it'll remain that simple - because then design decisions will not consider the endstate.
- samba/ganesha deployment, loud balancers to improve radosgw etc etc
Managing the access protocols/gateways is something we'll have to do. And again, that means we'll have to take HA into account. I'm a wee bit on the fence on the LB parts. We surely need to have hooks to inform the LB layer about which endpoints we deployed, but managing/deploying/configuring the LBs themselves I'd hope is out of scope.
- integration with platform management (why not show in the ceph dashboard whether you have patches outstanding against your host, or the host has a bad PSU) - enable the sysadmin to work more efficiently on Ceph, and maybe they'd prefer it over other platforms.
Why not? Because those are system management tasks that are outside Ceph. Please don't build an inferior salt/ansible/puppet/chef or systems management console. Solutions for these problems exist. Let's not duplicate everything.
</soapbox> +1
-1 Regards, Lars -- SUSE Software Solutions Germany GmbH, MD: Felix Imendörffer, HRB 36809 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
participants (6)
-
Anthony D'Atri
-
Jan Fajerski
-
Lars Marowsky-Bree
-
Paul Cuzner
-
Sage Weil
-
Sage Weil