[sepia] lab cluster status

Hi everyone, I realized today that I didn't tell everyone about the cephadm conversion on the lab cluster, so it seemed like a good time for an overall status update. ~3 weeks ago I converted the lab cluster to use cephadm. This helped shake out a number of issues with the upgrade process, and also ran into some exciting snags. The main issue was that cephadm only works with bluestore OSDs, and I had forgotten that there were lots of old filestore OSDs still in the cluster. To get around this, I ended up dist-ugprading several hosts from xenial to bionic so that the host packages could be installed (this all happened mid-upgrade and an upgrade bug was preventing the OSDs from peering). Once things had upgraded and stabilized, I removed most of the old mira nodes from the cluster (the ones that had all or mostly filestore OSDs) and rebalanced. Then I finished the cephadm conversion. Current status: - everything is cephadm and container-based - all OSDs are bluestore - there are 2 remaining mira nodes in the cluster - most of the hosts are still running xenial. One of the nice things about cephadm is that there are few OS dependencies--we just need podman or docker, python3, and LVM. So it's /mostly/ fine that these machines are running xenial. But it's not ideal. - We still have (and want) ceph-common on the host so that the ceph CLI works. But we don't build octopus for xenial, so the nautilus packages are still installed. There are a few new changes on the CLI side in octopus so we should fix this at some point. I think this just means we should dist-upgrade these machines to xenial. - There are still two mira in the cluster that we may want to remove at some point... A few other changes: - We have a few SSD-based OSDs, but the crush rules were still spreading data over everything. I updated the data pools to use hdd only and the cephfs metadata pool is now on SSDs only. This should have sped things up a bit! Cephadm stuff... For a crash course on cephadm, see the new docs at https://docs.ceph.com/docs/master/cephadm/ The main thing is that all of the daemons ar erunning in containers. If you're on the host and want to stop/start things, it's systemctl stop/start ceph-$fsid@$name For example, systemctl restart ceph-28f7427e-5558-4ffd-ae1a-51ec3042759a(a)mon.reesi002 (The nice thing is that tab completion works for the unit name.) There is also a ceph.target that will stop or start *all* ceph daemons, either for the cluster (ceph-28f7427e-5558-4ffd-ae1a-51ec3042759a.target) or all clusters (ceph.target). The cluster is configured to log to files like traditional ceph deploysments. The only difference is that logs are in /var/log/ceph/$fsid. Again, tab completion is your friend (esp when you remember that the fsid for this cluster starts with "2"). I've been upgrading this cluster regularly (almost every day, if not more) using the cephadm automated upgrades. That command is just ceph orch upgrade start --image quay.io/ceph-ci/ceph:octopus Cephadm can automatically decide where to deploy daemons based on specs you provide about placement, count, etc. You can view that with root@reesi002:~# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID alertmanager 1/1 9m ago 2d count:1 docker.io/prom/alertmanager:latest 0881eb8f169f crash 9/9 9m ago 2d * quay.io/ceph-ci/ceph:octopus a94ff4985406 grafana 1/1 67s ago 2d count:1 docker.io/pcuzner/ceph-grafana-el8:latest f77afcf0bcf6 mds.cephfs 4/4 9m ago 2d label:mds quay.io/ceph-ci/ceph:octopus d97834ddee42 mgr 2/2 7m ago 2d count:2 quay.io/ceph-ci/ceph:octopus a94ff4985406 mon 5/5 9m ago 2d count:5 quay.io/ceph-ci/ceph:octopus a94ff4985406 prometheus 1/1 9m ago 2d count:1 docker.io/prom/prometheus:latest e935122ab143 You can see the actual daemons with 'ceph orch ps'. You can see that the mds.cephfs service (cephfs == the fs name) is tied to label 'mds'. You can see host labels with root@reesi002:~# ceph orch host ls HOST ADDR LABELS STATUS mira055 mira055 mira060 mira060 mira093 mira093 reesi001 reesi001 mon mds reesi002 reesi002 mon mds reesi003 reesi003 mon mds reesi004 reesi004 mon mgr reesi005 reesi005 mon mgr reesi006 reesi006 mgr mds (There are other labels set there that aren't getting used at the moment.) TODOs... - dist-upgrade everything to xenial. - Upgrade the host packages. this is most easily done with ./cephadm add-repo --release octopus ./cephadm install cephadm ceph-common which will install the packaged cephadm (so it's in the path and doesn't have to be curled manually) and ceph-common (which has all the important CLI commands). We should probably uninstall the other ceph packages. - We aren't deploying the fulling monitoring (prometheus etc) stack via cephadm because some of those components are already installed (node-exporter I think?) and I'm not sure how that was done or the right way to remove them (or use them as is). Also at the moment there are a few bugs in the config files for alertmanager and grafana that cephadm is generating. sage