TL;DR: We could not fix this problem in the end and ended up with a Ceph fs in read only mode (so we could only backup, delete and restore) and one broken OSD (we deleted that and restored to a "new disk") I can now wrap up my whole experience with this problem. After the OSD usage growing to almost 2 TB x 3 OSDs (for a data that 'du' counted to be about 120GB), Ceph stopped filling up and in the week or two that followed, most of the space that was used showed up as free again. But there was one OSD that did not free up to meaningful values. To my surprise it was an OSD that is backed by SSDs not the one on HDD. It seems the biggest contributing factor was that I created the pools for the Ceph fs with autoscaling set to on (the default in cephadm dashboard). Now this pool never grew to more than 1 PG although it had a little over 100 GB on it. From what I read on this list this alone is prone to lock contention and other problematic behavior. Lessons learned: * If you use pool autoscaling, watch it if it actually does that. -> I opted for setting the new PG number for the replacement pools to 32 manually. * Ceph fs has a read only mode that at least lets you back up data in some bad states. -> That is good to know. It allows at least administrators to copy data to other storage devices. * If you use Ceph fs for persistent volumes in Kubernetes be aware that you probably loose all volumes at the same time when Ceph fs switches to read only. The CSI for Ceph fs does not work on read only Ceph fs, it always writes xattr on mount (the data pool that should be used and other internal data) and gives up if that fails. -> use a reasonable number of Ceph fs for Kubernetes persistent storage so you don't loose all PV at once. Other problems with my configuration I found: * We suffer from VMWare ESXi misreporting the type of disk physically attached. It reports some SAS SSDs as HDD. We also have real SAS HDD attached to some node. I suspected that to be a problem and we will exchange the HDDs for SSDs soon but the big problem was that the disktype was in the crushmap. -> Edited the crush map to ignore the type of storage as it is not very meaningful in our setup anyway * I had PGs stuck in undersized state for a long time and could not understand why ceph does not fix it. Then I checked the OSD weights (reweights) again and they were set to different values (1 and 0.85). After setting it to one on all OSDs Ceph actually started to bring all PG into the active+clean state. -> If all the OSDs actually are the same size i will either not reweight in the future or set it to the same value on all OSDs probably 1 So now after a noticable downtime of kubernetes and having to recreate most persistent volumes on the cluster Ceph health is HEALTH_OK again. I could upgrade to Ceph 16.2.13. I hope I can now upgrade to 17.2.6 without issues. Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam@oeaw.ac.at |www.oeaw.ac.at/acdh