I’m also using Rook on BM. I had never used K8s before, so that was the learning curve, e.g. translating the example YAML files into the Helm charts we needed, and the label / taint / toleration dance to fit the square peg of pinning services to round hole nodes. We’re using Kubespray ; I gather there are other ways of deploying K8s? Some things that could improve: * mgrs are limited to 2, apparently Sage previously said that was all anyone should need. I would like to be able to deploy one for each mon. * The efficiency of `destroy`ing OSDs is not exploited, so replacing one involves more data shuffling than it otherwise might * I’m specifying 3 RGWs but only getting 1 deployed, no idea why * Ingress / load balancer service for multiple RGWs seems to be manual * Bundled alerts are kind of noisy * I’m still unsure what Rook does dynamically, and what it only does at deployment time (we use ArgoCD). I.e., if I make changes, what sticks and what’s trampled? * How / if one can bake configuration (as in `ceph.conf` entries) into the YAML files vs manually running “ceph config” * What the sidecars within the pods are doing, if any of them can be disabled * Requests / limits for various pods, especially when on dedicated nodes. Plan to experiment with disabling limits and setting `autotune_memory_target_ratio` and `osd_memory_target_autotune` * Documentation for how to do pod-specific configuration, i.e. setting the number of OSDs per node when it isn’t uniform. A colleague helped me sort this out, but I’m enumerating each node - would like to be able to do so more concisely, perhaps with a default and overrides.
On Jul 6, 2023, at 4:13 AM, Joachim Kraftmayer - ceph ambassador <joachim.kraftmayer@clyso.com> wrote:
Hello
we have been following rook since 2018 and have had our experiences both on bare-metal and in the hyperscalers. In the same way, we have been following cephadm from the beginning.
Meanwhile, we have been using both in production for years and the decision which orchestrator to use depends from project to project. e.g., the features of both projects are not identical.
Joachim
___________________________________ ceph ambassador DACH ceph consultant since 2012
Clyso GmbH - Premier Ceph Foundation Member
Am 06.07.23 um 07:16 schrieb Nico Schottelius:
Morning,
we are running some ceph clusters with rook on bare metal and can very much recomend it. You should have proper k8s knowledge, knowing how to change objects such as configmaps or deployments, in case things go wrong.
In regards to stability, the rook operator is written rather defensive, not changing monitors or the cluster if the quorom is not met and checking how the osd status is on removal/adding of osds.
So TL;DR: very much usable and rather k8s native.
BR,
Nico
zssas@tuta.io writes:
Hello!
I am looking to simplify ceph management on bare-metal by deploying Rook onto kubernetes that has been deployed on bare metal (rke). I have used rook in a cloud environment but I have not used it on bare-metal. I am wondering if anyone here runs rook in bare-metal? Would you recommend it to cephadm or would you steer clear of it? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io