cephadm autotuning ceph-osd memory
Hi everyone, One of the things that ceph-ansible does that cephadm does not is automatically adjust the osd_memory_target based on the amount of memory and number of OSDs on the host. The approach is pretty simple: if the not_hci flag is set, then we take some factor (.7) * total memory and divide by osd count, and set osd_memory_target accordingly. We'd like to bring this "hands off" behavior over to cephadm in some form.. ideally one that is a bit more sophisticated. The currently proposal in written up in this pad: https://pad.ceph.com/p/autotune_memory_target The key ideas: - An autotune_memory_target config option (boolean) can be enabled for a specific daemon, class, host, or the whole cluster - It will apply not just to OSDs but other daemons. Currently mon and osd have a memory_target config; MDS has a similar setting for its cache that we can work with. The other daemons don't have configurable memory footprints (for the most part) but we can come up with some estimate of their *usage* based on some conservative estimates, maybe with cluster size factored in. - The autotuner would look at other (ceph) daemons on the host that aren't being tuned and deduct that memory from the total available, and then divvy up the remaining memory among the autotuned daemons. Initially we'd only support autotuning osd, mon, and mds. There are some open questions about how to implement some of the details, though. For instance, the current proposal doesn't record what the "tuned" memory is in ceph anywhere, but rather would look at what 'cephadm ls' says about the deployed containers' limits, compare that to what it thinks should happen, and redeploy daemons as needed. It might also make sense to allow these targets to be expressed via the 'service spec' yaml, which is more or less equivalent to the CRs in Rook (which currently do allow osd memory target to be set). I'm not entirely sure if this should be an orchestrator function (i.e., work with both rook and cephadm) or cephadm-only. It would be helpful to hear whether this sort of function would make any sense in a rook cluster... are there cases where the kubernetes nodes are dedicated to ceph and it makes sense to magically slurp up all of the hosts' RAM? We probably don't want to be in the business of checking with kubernetes about unused host resources and adjusting things whenever the k8s scheduler does something on the host... sage
On 2/11/21 5:11 PM, Sage Weil wrote:
Hi everyone,
One of the things that ceph-ansible does that cephadm does not is automatically adjust the osd_memory_target based on the amount of memory and number of OSDs on the host. The approach is pretty simple: if the not_hci flag is set, then we take some factor (.7) * total memory and divide by osd count, and set osd_memory_target accordingly.
We'd like to bring this "hands off" behavior over to cephadm in some form.. ideally one that is a bit more sophisticated. The currently proposal in written up in this pad:
https://pad.ceph.com/p/autotune_memory_target
The key ideas: - An autotune_memory_target config option (boolean) can be enabled for a specific daemon, class, host, or the whole cluster - It will apply not just to OSDs but other daemons. Currently mon and osd have a memory_target config; MDS has a similar setting for its cache that we can work with. The other daemons don't have configurable memory footprints (for the most part) but we can come up with some estimate of their *usage* based on some conservative estimates, maybe with cluster size factored in. - The autotuner would look at other (ceph) daemons on the host that aren't being tuned and deduct that memory from the total available, and then divvy up the remaining memory among the autotuned daemons. Initially we'd only support autotuning osd, mon, and mds.
There are some open questions about how to implement some of the details, though.
For instance, the current proposal doesn't record what the "tuned" memory is in ceph anywhere, but rather would look at what 'cephadm ls' says about the deployed containers' limits, compare that to what it thinks should happen, and redeploy daemons as needed.
It might also make sense to allow these targets to be expressed via the 'service spec' yaml, which is more or less equivalent to the CRs in Rook (which currently do allow osd memory target to be set).
I'm not entirely sure if this should be an orchestrator function (i.e., work with both rook and cephadm) or cephadm-only. It would be helpful to hear whether this sort of function would make any sense in a rook cluster... are there cases where the kubernetes nodes are dedicated to ceph and it makes sense to magically slurp up all of the hosts' RAM? We probably don't want to be in the business of checking with kubernetes about unused host resources and adjusting things whenever the k8s scheduler does something on the host...
sage
One question is what you do when you want to add more daemons to a node after the targets have already been set. An idea I've kicked around for bare-metal clusters is to let the daemons float their memory target between lower and upper bounds based on how close to a global memory target all daemons on a particular node are in aggregate. Each daemon would be constantly re-evaluating it's local target like we already do when determining the aggregate cache budget based on that memory target. One really neat thing you could with this model is set the aggressiveness of each daemon based on the priorities and sizes of unfulfilled allocation requests in the priority cache manager. The idea being that daemons that really aggressively want memory at high priorities can float their targets up and daemons that don't really care much can float their targets down. Ultimately all daemons would still be governed based on that global target though. (ie if we get close to or over that global target, even aggressive daemons would necessarily back off and reduce their targets). I suspect this can be done on bare metal without strictly requiring coordination in the mgr (though we could certainly do it that way too). I agree for containers. It's...different. I don't know if we can even really adjust container resource limits on the fly yet. We could set the container limits really high (which is basically what we already do, the memory_target is already way lower than the container limit). I don't know that we want to be trying to coordinate memory usage behind the containers back like what I proposed above. Ideally there would be some kind of rich api where something inside the container can negotiate for more (or less) memory for the container as a whole. I don't think anything like this exists though. Mark
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On Thu, Feb 11, 2021 at 7:47 PM Mark Nelson <mnelson@redhat.com> wrote:
On 2/11/21 5:11 PM, Sage Weil wrote:
Hi everyone,
One of the things that ceph-ansible does that cephadm does not is automatically adjust the osd_memory_target based on the amount of memory and number of OSDs on the host. The approach is pretty simple: if the not_hci flag is set, then we take some factor (.7) * total memory and divide by osd count, and set osd_memory_target accordingly.
We'd like to bring this "hands off" behavior over to cephadm in some form.. ideally one that is a bit more sophisticated. The currently proposal in written up in this pad:
Thank you. This would be very useful for the TripleO project's integration of cephadm [1] to deploy colocated OpenStack/Ceph. I understand you're still scoping the feature but can you say at this point if you think it would be available in Pacific? Thanks, John [1] https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/tripleo-ce...
The key ideas: - An autotune_memory_target config option (boolean) can be enabled for a specific daemon, class, host, or the whole cluster - It will apply not just to OSDs but other daemons. Currently mon and osd have a memory_target config; MDS has a similar setting for its cache that we can work with. The other daemons don't have configurable memory footprints (for the most part) but we can come up with some estimate of their *usage* based on some conservative estimates, maybe with cluster size factored in. - The autotuner would look at other (ceph) daemons on the host that aren't being tuned and deduct that memory from the total available, and then divvy up the remaining memory among the autotuned daemons. Initially we'd only support autotuning osd, mon, and mds.
There are some open questions about how to implement some of the details, though.
For instance, the current proposal doesn't record what the "tuned" memory is in ceph anywhere, but rather would look at what 'cephadm ls' says about the deployed containers' limits, compare that to what it thinks should happen, and redeploy daemons as needed.
It might also make sense to allow these targets to be expressed via the 'service spec' yaml, which is more or less equivalent to the CRs in Rook (which currently do allow osd memory target to be set).
I'm not entirely sure if this should be an orchestrator function (i.e., work with both rook and cephadm) or cephadm-only. It would be helpful to hear whether this sort of function would make any sense in a rook cluster... are there cases where the kubernetes nodes are dedicated to ceph and it makes sense to magically slurp up all of the hosts' RAM? We probably don't want to be in the business of checking with kubernetes about unused host resources and adjusting things whenever the k8s scheduler does something on the host...
sage
One question is what you do when you want to add more daemons to a node after the targets have already been set. An idea I've kicked around for bare-metal clusters is to let the daemons float their memory target between lower and upper bounds based on how close to a global memory target all daemons on a particular node are in aggregate. Each daemon would be constantly re-evaluating it's local target like we already do when determining the aggregate cache budget based on that memory target. One really neat thing you could with this model is set the aggressiveness of each daemon based on the priorities and sizes of unfulfilled allocation requests in the priority cache manager. The idea being that daemons that really aggressively want memory at high priorities can float their targets up and daemons that don't really care much can float their targets down. Ultimately all daemons would still be governed based on that global target though. (ie if we get close to or over that global target, even aggressive daemons would necessarily back off and reduce their targets). I suspect this can be done on bare metal without strictly requiring coordination in the mgr (though we could certainly do it that way too).
I agree for containers. It's...different. I don't know if we can even really adjust container resource limits on the fly yet. We could set the container limits really high (which is basically what we already do, the memory_target is already way lower than the container limit). I don't know that we want to be trying to coordinate memory usage behind the containers back like what I proposed above. Ideally there would be some kind of rich api where something inside the container can negotiate for more (or less) memory for the container as a whole. I don't think anything like this exists though.
Mark
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
I'm not sure if it will be in the initial pacific release but it will certainly get backported as soon as possible. On Fri, Feb 12, 2021 at 8:51 AM John Fulton <johfulto@redhat.com> wrote:
On Thu, Feb 11, 2021 at 7:47 PM Mark Nelson <mnelson@redhat.com> wrote:
On 2/11/21 5:11 PM, Sage Weil wrote:
Hi everyone,
One of the things that ceph-ansible does that cephadm does not is automatically adjust the osd_memory_target based on the amount of memory and number of OSDs on the host. The approach is pretty simple: if the not_hci flag is set, then we take some factor (.7) * total memory and divide by osd count, and set osd_memory_target accordingly.
We'd like to bring this "hands off" behavior over to cephadm in some form.. ideally one that is a bit more sophisticated. The currently proposal in written up in this pad:
Thank you. This would be very useful for the TripleO project's integration of cephadm [1] to deploy colocated OpenStack/Ceph.
I understand you're still scoping the feature but can you say at this point if you think it would be available in Pacific?
Thanks, John
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/tripleo-ce...
The key ideas: - An autotune_memory_target config option (boolean) can be enabled for a specific daemon, class, host, or the whole cluster - It will apply not just to OSDs but other daemons. Currently mon and osd have a memory_target config; MDS has a similar setting for its cache that we can work with. The other daemons don't have configurable memory footprints (for the most part) but we can come up with some estimate of their *usage* based on some conservative estimates, maybe with cluster size factored in. - The autotuner would look at other (ceph) daemons on the host that aren't being tuned and deduct that memory from the total available, and then divvy up the remaining memory among the autotuned daemons. Initially we'd only support autotuning osd, mon, and mds.
There are some open questions about how to implement some of the details, though.
For instance, the current proposal doesn't record what the "tuned" memory is in ceph anywhere, but rather would look at what 'cephadm ls' says about the deployed containers' limits, compare that to what it thinks should happen, and redeploy daemons as needed.
It might also make sense to allow these targets to be expressed via the 'service spec' yaml, which is more or less equivalent to the CRs in Rook (which currently do allow osd memory target to be set).
I'm not entirely sure if this should be an orchestrator function (i.e., work with both rook and cephadm) or cephadm-only. It would be helpful to hear whether this sort of function would make any sense in a rook cluster... are there cases where the kubernetes nodes are dedicated to ceph and it makes sense to magically slurp up all of the hosts' RAM? We probably don't want to be in the business of checking with kubernetes about unused host resources and adjusting things whenever the k8s scheduler does something on the host...
sage
One question is what you do when you want to add more daemons to a node after the targets have already been set. An idea I've kicked around for bare-metal clusters is to let the daemons float their memory target between lower and upper bounds based on how close to a global memory target all daemons on a particular node are in aggregate. Each daemon would be constantly re-evaluating it's local target like we already do when determining the aggregate cache budget based on that memory target. One really neat thing you could with this model is set the aggressiveness of each daemon based on the priorities and sizes of unfulfilled allocation requests in the priority cache manager. The idea being that daemons that really aggressively want memory at high priorities can float their targets up and daemons that don't really care much can float their targets down. Ultimately all daemons would still be governed based on that global target though. (ie if we get close to or over that global target, even aggressive daemons would necessarily back off and reduce their targets). I suspect this can be done on bare metal without strictly requiring coordination in the mgr (though we could certainly do it that way too).
I agree for containers. It's...different. I don't know if we can even really adjust container resource limits on the fly yet. We could set the container limits really high (which is basically what we already do, the memory_target is already way lower than the container limit). I don't know that we want to be trying to coordinate memory usage behind the containers back like what I proposed above. Ideally there would be some kind of rich api where something inside the container can negotiate for more (or less) memory for the container as a whole. I don't think anything like this exists though.
Mark
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
I left some notes on the pad - I think we'd want to treat mons a bit differently from other daemons, since more memory there isn't so important for performance, and perhaps scales with cluster size / osdmap churn rather than working set. Josh On 2/12/21 7:47 AM, Sage Weil wrote:
I'm not sure if it will be in the initial pacific release but it will certainly get backported as soon as possible.
On Fri, Feb 12, 2021 at 8:51 AM John Fulton <johfulto@redhat.com> wrote:
On Thu, Feb 11, 2021 at 7:47 PM Mark Nelson <mnelson@redhat.com> wrote:
On 2/11/21 5:11 PM, Sage Weil wrote:
Hi everyone,
One of the things that ceph-ansible does that cephadm does not is automatically adjust the osd_memory_target based on the amount of memory and number of OSDs on the host. The approach is pretty simple: if the not_hci flag is set, then we take some factor (.7) * total memory and divide by osd count, and set osd_memory_target accordingly.
We'd like to bring this "hands off" behavior over to cephadm in some form.. ideally one that is a bit more sophisticated. The currently proposal in written up in this pad:
Thank you. This would be very useful for the TripleO project's integration of cephadm [1] to deploy colocated OpenStack/Ceph.
I understand you're still scoping the feature but can you say at this point if you think it would be available in Pacific?
Thanks, John
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/tripleo-ce...
The key ideas: - An autotune_memory_target config option (boolean) can be enabled for a specific daemon, class, host, or the whole cluster - It will apply not just to OSDs but other daemons. Currently mon and osd have a memory_target config; MDS has a similar setting for its cache that we can work with. The other daemons don't have configurable memory footprints (for the most part) but we can come up with some estimate of their *usage* based on some conservative estimates, maybe with cluster size factored in. - The autotuner would look at other (ceph) daemons on the host that aren't being tuned and deduct that memory from the total available, and then divvy up the remaining memory among the autotuned daemons. Initially we'd only support autotuning osd, mon, and mds.
There are some open questions about how to implement some of the details, though.
For instance, the current proposal doesn't record what the "tuned" memory is in ceph anywhere, but rather would look at what 'cephadm ls' says about the deployed containers' limits, compare that to what it thinks should happen, and redeploy daemons as needed.
It might also make sense to allow these targets to be expressed via the 'service spec' yaml, which is more or less equivalent to the CRs in Rook (which currently do allow osd memory target to be set).
I'm not entirely sure if this should be an orchestrator function (i.e., work with both rook and cephadm) or cephadm-only. It would be helpful to hear whether this sort of function would make any sense in a rook cluster... are there cases where the kubernetes nodes are dedicated to ceph and it makes sense to magically slurp up all of the hosts' RAM? We probably don't want to be in the business of checking with kubernetes about unused host resources and adjusting things whenever the k8s scheduler does something on the host...
sage
One question is what you do when you want to add more daemons to a node after the targets have already been set. An idea I've kicked around for bare-metal clusters is to let the daemons float their memory target between lower and upper bounds based on how close to a global memory target all daemons on a particular node are in aggregate. Each daemon would be constantly re-evaluating it's local target like we already do when determining the aggregate cache budget based on that memory target. One really neat thing you could with this model is set the aggressiveness of each daemon based on the priorities and sizes of unfulfilled allocation requests in the priority cache manager. The idea being that daemons that really aggressively want memory at high priorities can float their targets up and daemons that don't really care much can float their targets down. Ultimately all daemons would still be governed based on that global target though. (ie if we get close to or over that global target, even aggressive daemons would necessarily back off and reduce their targets). I suspect this can be done on bare metal without strictly requiring coordination in the mgr (though we could certainly do it that way too).
I agree for containers. It's...different. I don't know if we can even really adjust container resource limits on the fly yet. We could set the container limits really high (which is basically what we already do, the memory_target is already way lower than the container limit). I don't know that we want to be trying to coordinate memory usage behind the containers back like what I proposed above. Ideally there would be some kind of rich api where something inside the container can negotiate for more (or less) memory for the container as a whole. I don't think anything like this exists though.
Mark
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
I suspect mon cache is not super important until something terrible happens and you've got an insane amount of rocksdb traffic where you really need the rocksdb block cache to avoid a ton of disk reads. I recall we've hit situations like that in the past. That might be a case where we would want the mon to be able to request "burst" memory at high priority for the block cache and lower the memory of any other daemons on the same node. As we talked about in the standup this morning this is difficult to do in the container world right now, but we could move toward a design that lets us do this on bare metal and potentially do it in containers if there's ever support for changing the container memory limits on the fly. Already we can change things like the osd_memory_target. It wouldn't be a stretch to let daemons float above or below that target based on relative priorities of requests in the priority cache manager imho. It's the just the container part that gets tricky. Mark On 2/12/21 10:05 AM, Josh Durgin wrote:
I left some notes on the pad - I think we'd want to treat mons a bit differently from other daemons, since more memory there isn't so important for performance, and perhaps scales with cluster size / osdmap churn rather than working set.
Josh
On 2/12/21 7:47 AM, Sage Weil wrote:
I'm not sure if it will be in the initial pacific release but it will certainly get backported as soon as possible.
On Fri, Feb 12, 2021 at 8:51 AM John Fulton <johfulto@redhat.com> wrote:
On Thu, Feb 11, 2021 at 7:47 PM Mark Nelson <mnelson@redhat.com> wrote:
On 2/11/21 5:11 PM, Sage Weil wrote:
Hi everyone,
One of the things that ceph-ansible does that cephadm does not is automatically adjust the osd_memory_target based on the amount of memory and number of OSDs on the host. The approach is pretty simple: if the not_hci flag is set, then we take some factor (.7) * total memory and divide by osd count, and set osd_memory_target accordingly.
We'd like to bring this "hands off" behavior over to cephadm in some form.. ideally one that is a bit more sophisticated. The currently proposal in written up in this pad:
Thank you. This would be very useful for the TripleO project's integration of cephadm [1] to deploy colocated OpenStack/Ceph.
I understand you're still scoping the feature but can you say at this point if you think it would be available in Pacific?
Thanks, John
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/tripleo-ce...
The key ideas: - An autotune_memory_target config option (boolean) can be enabled for a specific daemon, class, host, or the whole cluster - It will apply not just to OSDs but other daemons. Currently mon and osd have a memory_target config; MDS has a similar setting for its cache that we can work with. The other daemons don't have configurable memory footprints (for the most part) but we can come up with some estimate of their *usage* based on some conservative estimates, maybe with cluster size factored in. - The autotuner would look at other (ceph) daemons on the host that aren't being tuned and deduct that memory from the total available, and then divvy up the remaining memory among the autotuned daemons. Initially we'd only support autotuning osd, mon, and mds.
There are some open questions about how to implement some of the details, though.
For instance, the current proposal doesn't record what the "tuned" memory is in ceph anywhere, but rather would look at what 'cephadm ls' says about the deployed containers' limits, compare that to what it thinks should happen, and redeploy daemons as needed.
It might also make sense to allow these targets to be expressed via the 'service spec' yaml, which is more or less equivalent to the CRs in Rook (which currently do allow osd memory target to be set).
I'm not entirely sure if this should be an orchestrator function (i.e., work with both rook and cephadm) or cephadm-only. It would be helpful to hear whether this sort of function would make any sense in a rook cluster... are there cases where the kubernetes nodes are dedicated to ceph and it makes sense to magically slurp up all of the hosts' RAM? We probably don't want to be in the business of checking with kubernetes about unused host resources and adjusting things whenever the k8s scheduler does something on the host...
sage
One question is what you do when you want to add more daemons to a node after the targets have already been set. An idea I've kicked around for bare-metal clusters is to let the daemons float their memory target between lower and upper bounds based on how close to a global memory target all daemons on a particular node are in aggregate. Each daemon would be constantly re-evaluating it's local target like we already do when determining the aggregate cache budget based on that memory target. One really neat thing you could with this model is set the aggressiveness of each daemon based on the priorities and sizes of unfulfilled allocation requests in the priority cache manager. The idea being that daemons that really aggressively want memory at high priorities can float their targets up and daemons that don't really care much can float their targets down. Ultimately all daemons would still be governed based on that global target though. (ie if we get close to or over that global target, even aggressive daemons would necessarily back off and reduce their targets). I suspect this can be done on bare metal without strictly requiring coordination in the mgr (though we could certainly do it that way too).
I agree for containers. It's...different. I don't know if we can even really adjust container resource limits on the fly yet. We could set the container limits really high (which is basically what we already do, the memory_target is already way lower than the container limit). I don't know that we want to be trying to coordinate memory usage behind the containers back like what I proposed above. Ideally there would be some kind of rich api where something inside the container can negotiate for more (or less) memory for the container as a whole. I don't think anything like this exists though.
Mark
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Containers don't have to have a memory cgroup limit. It may be helpful to avoid that with cephadm, perhaps using a strict memory limit in testing but not in production to avoid potential availability problems. oomkilling is a bit drastic. Josh On 2/12/21 8:56 AM, Mark Nelson wrote:
I suspect mon cache is not super important until something terrible happens and you've got an insane amount of rocksdb traffic where you really need the rocksdb block cache to avoid a ton of disk reads. I recall we've hit situations like that in the past. That might be a case where we would want the mon to be able to request "burst" memory at high priority for the block cache and lower the memory of any other daemons on the same node. As we talked about in the standup this morning this is difficult to do in the container world right now, but we could move toward a design that lets us do this on bare metal and potentially do it in containers if there's ever support for changing the container memory limits on the fly. Already we can change things like the osd_memory_target. It wouldn't be a stretch to let daemons float above or below that target based on relative priorities of requests in the priority cache manager imho. It's the just the container part that gets tricky.
Mark
On 2/12/21 10:05 AM, Josh Durgin wrote:
I left some notes on the pad - I think we'd want to treat mons a bit differently from other daemons, since more memory there isn't so important for performance, and perhaps scales with cluster size / osdmap churn rather than working set.
Josh
On 2/12/21 7:47 AM, Sage Weil wrote:
I'm not sure if it will be in the initial pacific release but it will certainly get backported as soon as possible.
On Fri, Feb 12, 2021 at 8:51 AM John Fulton <johfulto@redhat.com> wrote:
On Thu, Feb 11, 2021 at 7:47 PM Mark Nelson <mnelson@redhat.com> wrote:
On 2/11/21 5:11 PM, Sage Weil wrote:
Hi everyone,
One of the things that ceph-ansible does that cephadm does not is automatically adjust the osd_memory_target based on the amount of memory and number of OSDs on the host. The approach is pretty simple: if the not_hci flag is set, then we take some factor (.7) * total memory and divide by osd count, and set osd_memory_target accordingly.
We'd like to bring this "hands off" behavior over to cephadm in some form.. ideally one that is a bit more sophisticated. The currently proposal in written up in this pad:
Thank you. This would be very useful for the TripleO project's integration of cephadm [1] to deploy colocated OpenStack/Ceph.
I understand you're still scoping the feature but can you say at this point if you think it would be available in Pacific?
Thanks, John
[1] https://specs.openstack.org/openstack/tripleo-specs/specs/wallaby/tripleo-ce...
The key ideas: - An autotune_memory_target config option (boolean) can be enabled for a specific daemon, class, host, or the whole cluster - It will apply not just to OSDs but other daemons. Currently mon and osd have a memory_target config; MDS has a similar setting for its cache that we can work with. The other daemons don't have configurable memory footprints (for the most part) but we can come up with some estimate of their *usage* based on some conservative estimates, maybe with cluster size factored in. - The autotuner would look at other (ceph) daemons on the host that aren't being tuned and deduct that memory from the total available, and then divvy up the remaining memory among the autotuned daemons. Initially we'd only support autotuning osd, mon, and mds.
There are some open questions about how to implement some of the details, though.
For instance, the current proposal doesn't record what the "tuned" memory is in ceph anywhere, but rather would look at what 'cephadm ls' says about the deployed containers' limits, compare that to what it thinks should happen, and redeploy daemons as needed.
It might also make sense to allow these targets to be expressed via the 'service spec' yaml, which is more or less equivalent to the CRs in Rook (which currently do allow osd memory target to be set).
I'm not entirely sure if this should be an orchestrator function (i.e., work with both rook and cephadm) or cephadm-only. It would be helpful to hear whether this sort of function would make any sense in a rook cluster... are there cases where the kubernetes nodes are dedicated to ceph and it makes sense to magically slurp up all of the hosts' RAM? We probably don't want to be in the business of checking with kubernetes about unused host resources and adjusting things whenever the k8s scheduler does something on the host...
sage
One question is what you do when you want to add more daemons to a node after the targets have already been set. An idea I've kicked around for bare-metal clusters is to let the daemons float their memory target between lower and upper bounds based on how close to a global memory target all daemons on a particular node are in aggregate. Each daemon would be constantly re-evaluating it's local target like we already do when determining the aggregate cache budget based on that memory target. One really neat thing you could with this model is set the aggressiveness of each daemon based on the priorities and sizes of unfulfilled allocation requests in the priority cache manager. The idea being that daemons that really aggressively want memory at high priorities can float their targets up and daemons that don't really care much can float their targets down. Ultimately all daemons would still be governed based on that global target though. (ie if we get close to or over that global target, even aggressive daemons would necessarily back off and reduce their targets). I suspect this can be done on bare metal without strictly requiring coordination in the mgr (though we could certainly do it that way too).
I agree for containers. It's...different. I don't know if we can even really adjust container resource limits on the fly yet. We could set the container limits really high (which is basically what we already do, the memory_target is already way lower than the container limit). I don't know that we want to be trying to coordinate memory usage behind the containers back like what I proposed above. Ideally there would be some kind of rich api where something inside the container can negotiate for more (or less) memory for the container as a whole. I don't think anything like this exists though.
Mark
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
_______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
On 2021-02-12T09:00:41, Josh Durgin <jdurgin@redhat.com> wrote:
Containers don't have to have a memory cgroup limit. It may be helpful to avoid that with cephadm, perhaps using a strict memory limit in testing but not in production to avoid potential availability problems.
Instead of handling this externally (and fairly statically?) via cephadm, how about the Ceph daemons on a node communicate to each other about memory availability/allocation/requirements dynamically? Then the "total" memory limit of Ceph on a node would be "per-pod". Ceph would manage the resources allocated to it itself. Say, going back to the point raised earlier, if an additional OSD is started, all others would reduce their cache targets dynamically to make room for that one (assuming there's enough space, otherwise the new daemon wouldn't fully spin up and exit/pause). The overall limits can even be enforced via the OS without k8s pods in cgroups if so chosen. Regards, Lars -- SUSE Software Solutions Germany GmbH, MD: Felix Imendörffer, HRB 36809 (AG Nürnberg) "Architects should open possibilities and not determine everything." (Ueli Zbinden)
On 2/16/21 4:29 AM, Lars Marowsky-Bree wrote:
On 2021-02-12T09:00:41, Josh Durgin <jdurgin@redhat.com> wrote:
Containers don't have to have a memory cgroup limit. It may be helpful to avoid that with cephadm, perhaps using a strict memory limit in testing but not in production to avoid potential availability problems. Instead of handling this externally (and fairly statically?) via cephadm, how about the Ceph daemons on a node communicate to each other about memory availability/allocation/requirements dynamically?
Then the "total" memory limit of Ceph on a node would be "per-pod". Ceph would manage the resources allocated to it itself.
Say, going back to the point raised earlier, if an additional OSD is started, all others would reduce their cache targets dynamically to make room for that one (assuming there's enough space, otherwise the new daemon wouldn't fully spin up and exit/pause).
The overall limits can even be enforced via the OS without k8s pods in cgroups if so chosen.
This is more or less the model I've had stewing on the backburner in my brain since kubecon in Barcelona. I'm not even sure we need very much communication to make it happen. Each daemon will know a global (node/pod/whatever) target (already shared via ceph.conf or pod or whatever). For individual memory usage calculations we can't really use RSS (We tried; It can lead to really bad thrashing because the kernel isn't guaranteed to release memory when we free it). For more global decisions we might be able to look at neighbor process info, though it's probably safer to keep using the current tcmalloc stats which we would have to share (but we already collect this in the perf counters for daemons using the prioritycachemanager, and we can update it fairly slower for this purpose). Ultimately each deamon would see not only it's local usage but the combined usage of all daemons in the same group. Each daemon would independently calculate how aggressive it wants to be given the fraction of the total it's already using and how close to the global limit all daemons are in aggregate. As the aggregate usage gets closer to the aggregate total, each daemon independently backs off at varying rates. The emergent behavior should be an aggregate backing off but with the individual rates based on local cache priorities combined with how close to the aggregate limit we are. As we get closer to the limit, some daemons would need to start freeing memory early to make up for daemons that are being more aggressive. Ultimately *all* daemons would need to back off when near (or above!) the target (Even if some do so earlier than others). As local priorities change we'd see small changes in allocations per daemon. To avoid this we could use the same "chunky" allocation scheme we use locally to avoid constant small allocation changes as relative aggressiveness changes slightly over time. None of this guarantees we always stay below the aggregate target (that's much harder given how our daemons work), but we could stay under the aggregate target with very high probability and hopefully minor spikes relative to the aggregate total. Mark
Regards, Lars
participants (6)
-
John Fulton
-
Josh Durgin
-
Lars Marowsky-Bree
-
Mark Nelson
-
Sage Weil
-
Sage Weil