Hi, After reading your posts for years, I feel compelled to ask for your help/advice. First, I need to explain the context of our CEPH cluster, the problems we have encountered, and finally, my questions. Thanks taking the time for reading me. Cheers, Olivier *** Background *** Our CEPH cluster was created in 2015 with version 0.94.x (Hammer), which has been upgraded over time to version 10.2.x (Jewel), then 12.x (Luminous) and then 14.x (Nautilus). The MONitors and CEPHstores have always run on Linux Debian, with versions updated according to the requirements for supporting the underlying hardware and/or CEPH releases. In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient. Each of the storage servers comes with HDDs whose size varies depending on the server generation, as well as SSDs whose size is more consistent but still varies (depending on price). The idea has always been to add HDD and SSD storage to the CEPH cluster when we add storage servers to expand it or replace old ones. At the OSD level, the basic rule has always been followed: one device = one OSD with metadatas (FileStore) on dedicated partitioned SSDs (up to 6 for 32 OSDs) and, for the past few years, on a partitioned NVMe RAID1 (MD). In total, we have: 100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB 84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.6 PiB 1.7 PiB 1.9 PiB 1.9 PiB 53.45 ssd 480 TiB 321 TiB 158 TiB 158 TiB 33.04 TOTAL 4.0 PiB 2.0 PiB 2.1 PiB 2.1 PiB 51.08 Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 3660.19751 root main_storage -11 1222.29883 datacenter DC1 -68 163.79984 host cephstore16 -280 109.09988 host cephstore28 -20 109.09988 host cephstore34 -289 109.09988 host cephstore31 -31 116.39990 host cephstore40 -205 116.39990 host cephstore37 -81 109.09988 host cephstore22 -71 163.79984 host cephstore19 -84 109.09988 host cephstore25 -179 116.39990 host cephstore43 -12 1222.29883 datacenter DC2 -69 163.79984 host cephstore17 -82 109.09988 host cephstore23 -295 109.09988 host cephstore32 -72 163.79984 host cephstore20 -283 109.09988 host cephstore29 -87 109.09988 host cephstore35 -85 109.09988 host cephstore26 -222 116.39990 host cephstore44 -36 116.39990 host cephstore41 -242 116.39990 host cephstore38 -25 1215.59998 datacenter DC3 -70 163.80000 host cephstore18 -74 163.80000 host cephstore21 -83 99.00000 host cephstore24 -86 110.00000 host cephstore27 -286 110.00000 host cephstore30 -298 99.00000 host cephstore33 -102 110.00000 host cephstore36 -304 120.00000 host cephstore39 -136 120.00000 host cephstore42 -307 120.00000 host cephstore45 -1 516.06305 root high-speed_storage -21 171.91544 datacenter xDC1 -62 16.84781 host xcephstore16 -259 14.00000 host xcephstore28 -3 14.00000 host xcephstore34 -268 14.00000 host xcephstore31 -310 14.30786 host xcephstore40 -105 14.30786 host xcephstore37 -46 30.68784 host xcephstore10 -75 11.67993 host xcephstore22 -61 16.09634 host xcephstore19 -78 11.67993 host xcephstore25 -322 14.30786 host xcephstore43 -15 171.16397 datacenter xDC2 -63 16.09634 host xcephstore17 -76 11.67993 host xcephstore23 -274 14.00000 host xcephstore32 -65 16.09634 host xcephstore20 -262 14.00000 host xcephstore29 -13 14.00000 host xcephstore35 -79 11.67993 host xcephstore26 -51 30.68784 host xcephstore11 -325 14.30786 host xcephstore44 -313 14.30786 host xcephstore41 -175 14.30786 host xcephstore38 -28 172.98364 datacenter xDC3 -56 30.68784 host xcephstore12 -64 16.09200 host xcephstore18 -67 16.09200 host xcephstore21 -77 12.00000 host xcephstore24 -80 12.00000 host xcephstore27 -265 14.39999 host xcephstore30 -277 14.39990 host xcephstore33 -17 14.39999 host xcephstore36 -204 14.30399 host xcephstore39 -319 14.30399 host xcephstore42 -328 14.30396 host xcephstore45 Our allocation rules are: # rules rule main_storage_ruleset { id 0 type replicated min_size 1 max_size 10 step take main_storage step chooseleaf firstn 0 type datacenter step emit } rule high-speed_storage_ruleset { id 1 type replicated min_size 1 max_size 10 step take high-speed_storage step chooseleaf firstn 0 type datacenter step emit } All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1). This CEPH cluster is currently only used for RBD. The volumes are used by our ~ 1,200 KVM VMs. *** Problems *** Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem). * First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start. We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore. * Second problem: Since our failure domain is a DC, we had to upgrade a DC and then wait for it to recover (~500 TB net). SSD storage recovery takes a few hours, while HDD storage recovery takes approximately three days. Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us. I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help). Once our VMs were shut down (crashed), the cluster completed its recovery (HDD-type OSDs) and, curiously, the SSD-type OSDs began to ‘empty’. The day after that, we began updating the storage servers in our second DC, and the phenomenon started again. We did not wait until we reached full_ratio to shut down our virtualisation environment and this time, the ‘SSD’ OSDs began to ‘empty’ after the following commands: ceph osd unset noscrub && ceph osd unset nodeep-scrub. In fact, we used to block scrubs and deep scrubs during massive upgrades and recoveries to save I/O. This never caused any problems in FileStore. It should be added that since we started using the CEPH Cluster (2015), scrubs have only been enabled at night so as not to impact production I/O, via the following options: osd_recovery_delay_start = 5, osd_scrub_begin_hour = 19, osd_scrub_end_hour = 7, osd_scrub_sleep = 0.1 (the latter may be removed since classes are now available) . After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer). *** Questions / Assumptions / Opinions *** Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution and is not desirable in the medium term, but we are dependent on recovery time (and, in addition, on the issue I am presenting to you here). Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ? I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case: root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’, This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up? (It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).) Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.