Upgrade CEPH 14.x -> 16.X + switch from Filestore to Bluestore = strange behavior
Hi, After reading your posts for years, I feel compelled to ask for your help/advice. First, I need to explain the context of our CEPH cluster, the problems we have encountered, and finally, my questions. Thanks taking the time for reading me. Cheers, Olivier *** Background *** Our CEPH cluster was created in 2015 with version 0.94.x (Hammer), which has been upgraded over time to version 10.2.x (Jewel), then 12.x (Luminous) and then 14.x (Nautilus). The MONitors and CEPHstores have always run on Linux Debian, with versions updated according to the requirements for supporting the underlying hardware and/or CEPH releases. In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient. Each of the storage servers comes with HDDs whose size varies depending on the server generation, as well as SSDs whose size is more consistent but still varies (depending on price). The idea has always been to add HDD and SSD storage to the CEPH cluster when we add storage servers to expand it or replace old ones. At the OSD level, the basic rule has always been followed: one device = one OSD with metadatas (FileStore) on dedicated partitioned SSDs (up to 6 for 32 OSDs) and, for the past few years, on a partitioned NVMe RAID1 (MD). In total, we have: 100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB 84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.6 PiB 1.7 PiB 1.9 PiB 1.9 PiB 53.45 ssd 480 TiB 321 TiB 158 TiB 158 TiB 33.04 TOTAL 4.0 PiB 2.0 PiB 2.1 PiB 2.1 PiB 51.08 Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 3660.19751 root main_storage -11 1222.29883 datacenter DC1 -68 163.79984 host cephstore16 -280 109.09988 host cephstore28 -20 109.09988 host cephstore34 -289 109.09988 host cephstore31 -31 116.39990 host cephstore40 -205 116.39990 host cephstore37 -81 109.09988 host cephstore22 -71 163.79984 host cephstore19 -84 109.09988 host cephstore25 -179 116.39990 host cephstore43 -12 1222.29883 datacenter DC2 -69 163.79984 host cephstore17 -82 109.09988 host cephstore23 -295 109.09988 host cephstore32 -72 163.79984 host cephstore20 -283 109.09988 host cephstore29 -87 109.09988 host cephstore35 -85 109.09988 host cephstore26 -222 116.39990 host cephstore44 -36 116.39990 host cephstore41 -242 116.39990 host cephstore38 -25 1215.59998 datacenter DC3 -70 163.80000 host cephstore18 -74 163.80000 host cephstore21 -83 99.00000 host cephstore24 -86 110.00000 host cephstore27 -286 110.00000 host cephstore30 -298 99.00000 host cephstore33 -102 110.00000 host cephstore36 -304 120.00000 host cephstore39 -136 120.00000 host cephstore42 -307 120.00000 host cephstore45 -1 516.06305 root high-speed_storage -21 171.91544 datacenter xDC1 -62 16.84781 host xcephstore16 -259 14.00000 host xcephstore28 -3 14.00000 host xcephstore34 -268 14.00000 host xcephstore31 -310 14.30786 host xcephstore40 -105 14.30786 host xcephstore37 -46 30.68784 host xcephstore10 -75 11.67993 host xcephstore22 -61 16.09634 host xcephstore19 -78 11.67993 host xcephstore25 -322 14.30786 host xcephstore43 -15 171.16397 datacenter xDC2 -63 16.09634 host xcephstore17 -76 11.67993 host xcephstore23 -274 14.00000 host xcephstore32 -65 16.09634 host xcephstore20 -262 14.00000 host xcephstore29 -13 14.00000 host xcephstore35 -79 11.67993 host xcephstore26 -51 30.68784 host xcephstore11 -325 14.30786 host xcephstore44 -313 14.30786 host xcephstore41 -175 14.30786 host xcephstore38 -28 172.98364 datacenter xDC3 -56 30.68784 host xcephstore12 -64 16.09200 host xcephstore18 -67 16.09200 host xcephstore21 -77 12.00000 host xcephstore24 -80 12.00000 host xcephstore27 -265 14.39999 host xcephstore30 -277 14.39990 host xcephstore33 -17 14.39999 host xcephstore36 -204 14.30399 host xcephstore39 -319 14.30399 host xcephstore42 -328 14.30396 host xcephstore45 Our allocation rules are: # rules rule main_storage_ruleset { id 0 type replicated min_size 1 max_size 10 step take main_storage step chooseleaf firstn 0 type datacenter step emit } rule high-speed_storage_ruleset { id 1 type replicated min_size 1 max_size 10 step take high-speed_storage step chooseleaf firstn 0 type datacenter step emit } All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1). This CEPH cluster is currently only used for RBD. The volumes are used by our ~ 1,200 KVM VMs. *** Problems *** Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem). * First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start. We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore. * Second problem: Since our failure domain is a DC, we had to upgrade a DC and then wait for it to recover (~500 TB net). SSD storage recovery takes a few hours, while HDD storage recovery takes approximately three days. Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us. I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help). Once our VMs were shut down (crashed), the cluster completed its recovery (HDD-type OSDs) and, curiously, the SSD-type OSDs began to ‘empty’. The day after that, we began updating the storage servers in our second DC, and the phenomenon started again. We did not wait until we reached full_ratio to shut down our virtualisation environment and this time, the ‘SSD’ OSDs began to ‘empty’ after the following commands: ceph osd unset noscrub && ceph osd unset nodeep-scrub. In fact, we used to block scrubs and deep scrubs during massive upgrades and recoveries to save I/O. This never caused any problems in FileStore. It should be added that since we started using the CEPH Cluster (2015), scrubs have only been enabled at night so as not to impact production I/O, via the following options: osd_recovery_delay_start = 5, osd_scrub_begin_hour = 19, osd_scrub_end_hour = 7, osd_scrub_sleep = 0.1 (the latter may be removed since classes are now available) . After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer). *** Questions / Assumptions / Opinions *** Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution and is not desirable in the medium term, but we are dependent on recovery time (and, in addition, on the issue I am presenting to you here). Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ? I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case: root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’, This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up? (It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).) Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.
Hi, I can't say that we upgraded a lot of clusters from N to P, but those that we upgraded didn't show any of these symptoms you describe. But we always did the Filestore to Bluestore conversion before the actual upgrade. In SUSE Enterprise Storage (which we also supported at that time) this was pointed out as a requirement. I just checked the ceph docs, I can't find such a statement (yet).
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
I would recommend to increase min_size to 2, otherwise you let Ceph lose two of three PGs before pausing IO, this can make recovery difficult. Reducing min_size to 1 should only be a temporary solution to preserve stalling client IO during recovery. Regards, Eugen Zitat von Olivier Delcourt <olivier.delcourt@uclouvain.be>:
Hi,
After reading your posts for years, I feel compelled to ask for your help/advice. First, I need to explain the context of our CEPH cluster, the problems we have encountered, and finally, my questions. Thanks taking the time for reading me. Cheers, Olivier
*** Background ***
Our CEPH cluster was created in 2015 with version 0.94.x (Hammer), which has been upgraded over time to version 10.2.x (Jewel), then 12.x (Luminous) and then 14.x (Nautilus). The MONitors and CEPHstores have always run on Linux Debian, with versions updated according to the requirements for supporting the underlying hardware and/or CEPH releases.
In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient.
Each of the storage servers comes with HDDs whose size varies depending on the server generation, as well as SSDs whose size is more consistent but still varies (depending on price). The idea has always been to add HDD and SSD storage to the CEPH cluster when we add storage servers to expand it or replace old ones. At the OSD level, the basic rule has always been followed: one device = one OSD with metadatas (FileStore) on dedicated partitioned SSDs (up to 6 for 32 OSDs) and, for the past few years, on a partitioned NVMe RAID1 (MD).
In total, we have:
100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB
84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.6 PiB 1.7 PiB 1.9 PiB 1.9 PiB 53.45 ssd 480 TiB 321 TiB 158 TiB 158 TiB 33.04 TOTAL 4.0 PiB 2.0 PiB 2.1 PiB 2.1 PiB 51.08
Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 3660.19751 root main_storage -11 1222.29883 datacenter DC1 -68 163.79984 host cephstore16 -280 109.09988 host cephstore28 -20 109.09988 host cephstore34 -289 109.09988 host cephstore31 -31 116.39990 host cephstore40 -205 116.39990 host cephstore37 -81 109.09988 host cephstore22 -71 163.79984 host cephstore19 -84 109.09988 host cephstore25 -179 116.39990 host cephstore43 -12 1222.29883 datacenter DC2 -69 163.79984 host cephstore17 -82 109.09988 host cephstore23 -295 109.09988 host cephstore32 -72 163.79984 host cephstore20 -283 109.09988 host cephstore29 -87 109.09988 host cephstore35 -85 109.09988 host cephstore26 -222 116.39990 host cephstore44 -36 116.39990 host cephstore41 -242 116.39990 host cephstore38 -25 1215.59998 datacenter DC3 -70 163.80000 host cephstore18 -74 163.80000 host cephstore21 -83 99.00000 host cephstore24 -86 110.00000 host cephstore27 -286 110.00000 host cephstore30 -298 99.00000 host cephstore33 -102 110.00000 host cephstore36 -304 120.00000 host cephstore39 -136 120.00000 host cephstore42 -307 120.00000 host cephstore45
-1 516.06305 root high-speed_storage -21 171.91544 datacenter xDC1 -62 16.84781 host xcephstore16 -259 14.00000 host xcephstore28 -3 14.00000 host xcephstore34 -268 14.00000 host xcephstore31 -310 14.30786 host xcephstore40 -105 14.30786 host xcephstore37 -46 30.68784 host xcephstore10 -75 11.67993 host xcephstore22 -61 16.09634 host xcephstore19 -78 11.67993 host xcephstore25 -322 14.30786 host xcephstore43 -15 171.16397 datacenter xDC2 -63 16.09634 host xcephstore17 -76 11.67993 host xcephstore23 -274 14.00000 host xcephstore32 -65 16.09634 host xcephstore20 -262 14.00000 host xcephstore29 -13 14.00000 host xcephstore35 -79 11.67993 host xcephstore26 -51 30.68784 host xcephstore11 -325 14.30786 host xcephstore44 -313 14.30786 host xcephstore41 -175 14.30786 host xcephstore38 -28 172.98364 datacenter xDC3 -56 30.68784 host xcephstore12 -64 16.09200 host xcephstore18 -67 16.09200 host xcephstore21 -77 12.00000 host xcephstore24 -80 12.00000 host xcephstore27 -265 14.39999 host xcephstore30 -277 14.39990 host xcephstore33 -17 14.39999 host xcephstore36 -204 14.30399 host xcephstore39 -319 14.30399 host xcephstore42 -328 14.30396 host xcephstore45
Our allocation rules are:
# rules rule main_storage_ruleset { id 0 type replicated min_size 1 max_size 10 step take main_storage step chooseleaf firstn 0 type datacenter step emit } rule high-speed_storage_ruleset { id 1 type replicated min_size 1 max_size 10 step take high-speed_storage step chooseleaf firstn 0 type datacenter step emit }
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
This CEPH cluster is currently only used for RBD. The volumes are used by our ~ 1,200 KVM VMs.
*** Problems ***
Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem).
* First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start. We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore.
* Second problem: Since our failure domain is a DC, we had to upgrade a DC and then wait for it to recover (~500 TB net). SSD storage recovery takes a few hours, while HDD storage recovery takes approximately three days. Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us. I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help). Once our VMs were shut down (crashed), the cluster completed its recovery (HDD-type OSDs) and, curiously, the SSD-type OSDs began to ‘empty’.
The day after that, we began updating the storage servers in our second DC, and the phenomenon started again. We did not wait until we reached full_ratio to shut down our virtualisation environment and this time, the ‘SSD’ OSDs began to ‘empty’ after the following commands: ceph osd unset noscrub && ceph osd unset nodeep-scrub.
In fact, we used to block scrubs and deep scrubs during massive upgrades and recoveries to save I/O. This never caused any problems in FileStore. It should be added that since we started using the CEPH Cluster (2015), scrubs have only been enabled at night so as not to impact production I/O, via the following options: osd_recovery_delay_start = 5, osd_scrub_begin_hour = 19, osd_scrub_end_hour = 7, osd_scrub_sleep = 0.1 (the latter may be removed since classes are now available) .
After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer).
*** Questions / Assumptions / Opinions ***
Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution and is not desirable in the medium term, but we are dependent on recovery time (and, in addition, on the issue I am presenting to you here).
Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ?
I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case:
root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’,
This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up?
(It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).)
Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.
Hi Oliver, I agree with what Eugen wrote. From Hammer to the Pacific version is already a long journey. Across all major versions, I have seen and experienced many challenges and major changes. Some of these changes only became apparent years later. I have seen the behavior you describe a few times, but it can have many causes. Regards, Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Sa., 13. Sept. 2025 um 08:45 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi,
I can't say that we upgraded a lot of clusters from N to P, but those that we upgraded didn't show any of these symptoms you describe. But we always did the Filestore to Bluestore conversion before the actual upgrade. In SUSE Enterprise Storage (which we also supported at that time) this was pointed out as a requirement. I just checked the ceph docs, I can't find such a statement (yet).
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
I would recommend to increase min_size to 2, otherwise you let Ceph lose two of three PGs before pausing IO, this can make recovery difficult. Reducing min_size to 1 should only be a temporary solution to preserve stalling client IO during recovery.
Regards, Eugen
Zitat von Olivier Delcourt <olivier.delcourt@uclouvain.be>:
Hi,
After reading your posts for years, I feel compelled to ask for your help/advice. First, I need to explain the context of our CEPH cluster, the problems we have encountered, and finally, my questions. Thanks taking the time for reading me. Cheers, Olivier
*** Background ***
Our CEPH cluster was created in 2015 with version 0.94.x (Hammer), which has been upgraded over time to version 10.2.x (Jewel), then 12.x (Luminous) and then 14.x (Nautilus). The MONitors and CEPHstores have always run on Linux Debian, with versions updated according to the requirements for supporting the underlying hardware and/or CEPH releases.
In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient.
Each of the storage servers comes with HDDs whose size varies depending on the server generation, as well as SSDs whose size is more consistent but still varies (depending on price). The idea has always been to add HDD and SSD storage to the CEPH cluster when we add storage servers to expand it or replace old ones. At the OSD level, the basic rule has always been followed: one device = one OSD with metadatas (FileStore) on dedicated partitioned SSDs (up to 6 for 32 OSDs) and, for the past few years, on a partitioned NVMe RAID1 (MD).
In total, we have:
100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB
84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.6 PiB 1.7 PiB 1.9 PiB 1.9 PiB 53.45 ssd 480 TiB 321 TiB 158 TiB 158 TiB 33.04 TOTAL 4.0 PiB 2.0 PiB 2.1 PiB 2.1 PiB 51.08
Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 3660.19751 root main_storage -11 1222.29883 datacenter DC1 -68 163.79984 host cephstore16 -280 109.09988 host cephstore28 -20 109.09988 host cephstore34 -289 109.09988 host cephstore31 -31 116.39990 host cephstore40 -205 116.39990 host cephstore37 -81 109.09988 host cephstore22 -71 163.79984 host cephstore19 -84 109.09988 host cephstore25 -179 116.39990 host cephstore43 -12 1222.29883 datacenter DC2 -69 163.79984 host cephstore17 -82 109.09988 host cephstore23 -295 109.09988 host cephstore32 -72 163.79984 host cephstore20 -283 109.09988 host cephstore29 -87 109.09988 host cephstore35 -85 109.09988 host cephstore26 -222 116.39990 host cephstore44 -36 116.39990 host cephstore41 -242 116.39990 host cephstore38 -25 1215.59998 datacenter DC3 -70 163.80000 host cephstore18 -74 163.80000 host cephstore21 -83 99.00000 host cephstore24 -86 110.00000 host cephstore27 -286 110.00000 host cephstore30 -298 99.00000 host cephstore33 -102 110.00000 host cephstore36 -304 120.00000 host cephstore39 -136 120.00000 host cephstore42 -307 120.00000 host cephstore45
-1 516.06305 root high-speed_storage -21 171.91544 datacenter xDC1 -62 16.84781 host xcephstore16 -259 14.00000 host xcephstore28 -3 14.00000 host xcephstore34 -268 14.00000 host xcephstore31 -310 14.30786 host xcephstore40 -105 14.30786 host xcephstore37 -46 30.68784 host xcephstore10 -75 11.67993 host xcephstore22 -61 16.09634 host xcephstore19 -78 11.67993 host xcephstore25 -322 14.30786 host xcephstore43 -15 171.16397 datacenter xDC2 -63 16.09634 host xcephstore17 -76 11.67993 host xcephstore23 -274 14.00000 host xcephstore32 -65 16.09634 host xcephstore20 -262 14.00000 host xcephstore29 -13 14.00000 host xcephstore35 -79 11.67993 host xcephstore26 -51 30.68784 host xcephstore11 -325 14.30786 host xcephstore44 -313 14.30786 host xcephstore41 -175 14.30786 host xcephstore38 -28 172.98364 datacenter xDC3 -56 30.68784 host xcephstore12 -64 16.09200 host xcephstore18 -67 16.09200 host xcephstore21 -77 12.00000 host xcephstore24 -80 12.00000 host xcephstore27 -265 14.39999 host xcephstore30 -277 14.39990 host xcephstore33 -17 14.39999 host xcephstore36 -204 14.30399 host xcephstore39 -319 14.30399 host xcephstore42 -328 14.30396 host xcephstore45
Our allocation rules are:
# rules rule main_storage_ruleset { id 0 type replicated min_size 1 max_size 10 step take main_storage step chooseleaf firstn 0 type datacenter step emit } rule high-speed_storage_ruleset { id 1 type replicated min_size 1 max_size 10 step take high-speed_storage step chooseleaf firstn 0 type datacenter step emit }
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
This CEPH cluster is currently only used for RBD. The volumes are used by our ~ 1,200 KVM VMs.
*** Problems ***
Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem).
* First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start. We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore.
* Second problem: Since our failure domain is a DC, we had to upgrade a DC and then wait for it to recover (~500 TB net). SSD storage recovery takes a few hours, while HDD storage recovery takes approximately three days. Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us. I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help). Once our VMs were shut down (crashed), the cluster completed its recovery (HDD-type OSDs) and, curiously, the SSD-type OSDs began to ‘empty’.
The day after that, we began updating the storage servers in our second DC, and the phenomenon started again. We did not wait until we reached full_ratio to shut down our virtualisation environment and this time, the ‘SSD’ OSDs began to ‘empty’ after the following commands: ceph osd unset noscrub && ceph osd unset nodeep-scrub.
In fact, we used to block scrubs and deep scrubs during massive upgrades and recoveries to save I/O. This never caused any problems in FileStore. It should be added that since we started using the CEPH Cluster (2015), scrubs have only been enabled at night so as not to impact production I/O, via the following options: osd_recovery_delay_start = 5, osd_scrub_begin_hour = 19, osd_scrub_end_hour = 7, osd_scrub_sleep = 0.1 (the latter may be removed since classes are now available) .
After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer).
*** Questions / Assumptions / Opinions ***
Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution and is not desirable in the medium term, but we are dependent on recovery time (and, in addition, on the issue I am presenting to you here).
Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ?
I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case:
root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’,
This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up?
(It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).)
Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Olivier, Not sure it really helps but I wanted to say that our main cluster has the same history as yours, we did this long journey moving in a few months from Jewell to octopus, and, conversely to Joachim, we experienced no major issue ( we are currently running Reef on this cluster) following carefully the release notes for each version. The one difference is that we did the Filestore to Bluestore conversation in the last version said to support Filestore, Nautilus if I am right. We also switched to Ceph and during Octopus. We also have the same zoology of OSD and I wouldn't stay it is a problem in itself, even though it may require some adjustments to properly balance PG. The balancer is pretty good at this. Another difference is that we never splitter data and metadata on OSD. I think the main recommendations to move forward has been given by Eugene and Joachim... Good luck. Michel Sent from my mobile Le 13 septembre 2025 11:00:10 Joachim Kraftmayer <joachim.kraftmayer@clyso.com> a écrit :
Hi Oliver,
I agree with what Eugen wrote. From Hammer to the Pacific version is already a long journey. Across all major versions, I have seen and experienced many challenges and major changes. Some of these changes only became apparent years later. I have seen the behavior you describe a few times, but it can have many causes. Regards, Joachim
joachim.kraftmayer@clyso.com
www.clyso.com
Hohenzollernstr. 27, 80801 Munich
Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677
Am Sa., 13. Sept. 2025 um 08:45 Uhr schrieb Eugen Block <eblock@nde.ag>:
Hi,
I can't say that we upgraded a lot of clusters from N to P, but those that we upgraded didn't show any of these symptoms you describe. But we always did the Filestore to Bluestore conversion before the actual upgrade. In SUSE Enterprise Storage (which we also supported at that time) this was pointed out as a requirement. I just checked the ceph docs, I can't find such a statement (yet).
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
I would recommend to increase min_size to 2, otherwise you let Ceph lose two of three PGs before pausing IO, this can make recovery difficult. Reducing min_size to 1 should only be a temporary solution to preserve stalling client IO during recovery.
Regards, Eugen
Zitat von Olivier Delcourt <olivier.delcourt@uclouvain.be>:
Hi,
After reading your posts for years, I feel compelled to ask for your help/advice. First, I need to explain the context of our CEPH cluster, the problems we have encountered, and finally, my questions. Thanks taking the time for reading me. Cheers, Olivier
*** Background ***
Our CEPH cluster was created in 2015 with version 0.94.x (Hammer), which has been upgraded over time to version 10.2.x (Jewel), then 12.x (Luminous) and then 14.x (Nautilus). The MONitors and CEPHstores have always run on Linux Debian, with versions updated according to the requirements for supporting the underlying hardware and/or CEPH releases.
In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient.
Each of the storage servers comes with HDDs whose size varies depending on the server generation, as well as SSDs whose size is more consistent but still varies (depending on price). The idea has always been to add HDD and SSD storage to the CEPH cluster when we add storage servers to expand it or replace old ones. At the OSD level, the basic rule has always been followed: one device = one OSD with metadatas (FileStore) on dedicated partitioned SSDs (up to 6 for 32 OSDs) and, for the past few years, on a partitioned NVMe RAID1 (MD).
In total, we have:
100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB
84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB
--- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.6 PiB 1.7 PiB 1.9 PiB 1.9 PiB 53.45 ssd 480 TiB 321 TiB 158 TiB 158 TiB 33.04 TOTAL 4.0 PiB 2.0 PiB 2.1 PiB 2.1 PiB 51.08
Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -2 3660.19751 root main_storage -11 1222.29883 datacenter DC1 -68 163.79984 host cephstore16 -280 109.09988 host cephstore28 -20 109.09988 host cephstore34 -289 109.09988 host cephstore31 -31 116.39990 host cephstore40 -205 116.39990 host cephstore37 -81 109.09988 host cephstore22 -71 163.79984 host cephstore19 -84 109.09988 host cephstore25 -179 116.39990 host cephstore43 -12 1222.29883 datacenter DC2 -69 163.79984 host cephstore17 -82 109.09988 host cephstore23 -295 109.09988 host cephstore32 -72 163.79984 host cephstore20 -283 109.09988 host cephstore29 -87 109.09988 host cephstore35 -85 109.09988 host cephstore26 -222 116.39990 host cephstore44 -36 116.39990 host cephstore41 -242 116.39990 host cephstore38 -25 1215.59998 datacenter DC3 -70 163.80000 host cephstore18 -74 163.80000 host cephstore21 -83 99.00000 host cephstore24 -86 110.00000 host cephstore27 -286 110.00000 host cephstore30 -298 99.00000 host cephstore33 -102 110.00000 host cephstore36 -304 120.00000 host cephstore39 -136 120.00000 host cephstore42 -307 120.00000 host cephstore45
-1 516.06305 root high-speed_storage -21 171.91544 datacenter xDC1 -62 16.84781 host xcephstore16 -259 14.00000 host xcephstore28 -3 14.00000 host xcephstore34 -268 14.00000 host xcephstore31 -310 14.30786 host xcephstore40 -105 14.30786 host xcephstore37 -46 30.68784 host xcephstore10 -75 11.67993 host xcephstore22 -61 16.09634 host xcephstore19 -78 11.67993 host xcephstore25 -322 14.30786 host xcephstore43 -15 171.16397 datacenter xDC2 -63 16.09634 host xcephstore17 -76 11.67993 host xcephstore23 -274 14.00000 host xcephstore32 -65 16.09634 host xcephstore20 -262 14.00000 host xcephstore29 -13 14.00000 host xcephstore35 -79 11.67993 host xcephstore26 -51 30.68784 host xcephstore11 -325 14.30786 host xcephstore44 -313 14.30786 host xcephstore41 -175 14.30786 host xcephstore38 -28 172.98364 datacenter xDC3 -56 30.68784 host xcephstore12 -64 16.09200 host xcephstore18 -67 16.09200 host xcephstore21 -77 12.00000 host xcephstore24 -80 12.00000 host xcephstore27 -265 14.39999 host xcephstore30 -277 14.39990 host xcephstore33 -17 14.39999 host xcephstore36 -204 14.30399 host xcephstore39 -319 14.30399 host xcephstore42 -328 14.30396 host xcephstore45
Our allocation rules are:
# rules rule main_storage_ruleset { id 0 type replicated min_size 1 max_size 10 step take main_storage step chooseleaf firstn 0 type datacenter step emit } rule high-speed_storage_ruleset { id 1 type replicated min_size 1 max_size 10 step take high-speed_storage step chooseleaf firstn 0 type datacenter step emit }
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
This CEPH cluster is currently only used for RBD. The volumes are used by our ~ 1,200 KVM VMs.
*** Problems ***
Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem).
* First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start. We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore.
* Second problem: Since our failure domain is a DC, we had to upgrade a DC and then wait for it to recover (~500 TB net). SSD storage recovery takes a few hours, while HDD storage recovery takes approximately three days. Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us. I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help). Once our VMs were shut down (crashed), the cluster completed its recovery (HDD-type OSDs) and, curiously, the SSD-type OSDs began to ‘empty’.
The day after that, we began updating the storage servers in our second DC, and the phenomenon started again. We did not wait until we reached full_ratio to shut down our virtualisation environment and this time, the ‘SSD’ OSDs began to ‘empty’ after the following commands: ceph osd unset noscrub && ceph osd unset nodeep-scrub.
In fact, we used to block scrubs and deep scrubs during massive upgrades and recoveries to save I/O. This never caused any problems in FileStore. It should be added that since we started using the CEPH Cluster (2015), scrubs have only been enabled at night so as not to impact production I/O, via the following options: osd_recovery_delay_start = 5, osd_scrub_begin_hour = 19, osd_scrub_end_hour = 7, osd_scrub_sleep = 0.1 (the latter may be removed since classes are now available) .
After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer).
*** Questions / Assumptions / Opinions ***
Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution and is not desirable in the medium term, but we are dependent on recovery time (and, in addition, on the issue I am presenting to you here).
Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ?
I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case:
root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’,
This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up?
(It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).)
Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
From Hammer to the Pacific version is already a long journey.
I've seen some odd mon behavior from clusters initially installed on Hammer, but that surfaced in Jewel and is I doubt in play here.
I can't say that we upgraded a lot of clusters from N to P, but those that we upgraded didn't show any of these symptoms you describe. But we always did the Filestore to Bluestore conversion before the actual upgrade. In SUSE Enterprise Storage (which we also supported at that time) this was pointed out as a requirement. I just checked the ceph docs, I can't find such a statement (yet).
I *think* Filestore OSDs still work, but I've been peppering the docs with admonitions to convert for several releases now. I would expect them to have worked with Pacific. Did you update from the last Nautilus to the last Pacific?
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
I would recommend to increase min_size to 2, otherwise you let Ceph lose two of three PGs before pausing IO, this can make recovery difficult. Reducing min_size to 1 should only be a temporary solution to preserve stalling client IO during recovery.
Absolutely.
. First, I need to explain the context of our CEPH
Ceph, please. Not CEPH. ;)
In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient.
Nice planning. For scratch clusters today I usually suggest a single 25-100 GE bonded public network. The dynamics when you started were different.
Each of the storage servers comes with HDDs whose size varies depending on the server generation
Good job keeping each DC's aggregate CRUSH weight nearly identical.
as well as SSDs whose size is more consistent but still varies (depending on price).
Enterprise SAS/SATA SSDs are starting to disappear from the market. The next time you buy servers, consider NVMe-only chassis. With careful procurement and by not having to pay for an HBA, they can be more affordable than you might think, and conserve precious RUs. And larger SSDs get you more capacity per chassis, so you can save on chassis, switch ports, etc.
100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB
Have you tried disabling their volatile write cache?
84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB
Just for others reading, very small SSDs can end up using a surprising fraction of their capacity for DB/WAL/other overhead, resulting in less usable capacity than one expects.
Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees:
Indeed, that was a common strategy. Device classes are way more convenient and work better with the upmap balancer. At some point you might consider converting with the crush tool reclassify feature, see: https://docs.ceph.com/en/pacific/rados/operations/crush-map-edits/
Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem).
I'm not experienced with Debian as such, but Filestore OSDs being XFS filesystems are susceptible to XFS flern, which means both the kernel and filesystem utilities. I've seen [non-Ceph] issues with a large kernel version jump. In those situations a one-time xfs_repair addressed the problem. I don't recall the details, but ISTR that at a certain point XFS started paying closer attention to a certain filesystem structure than it used to, so older filesystems that previously ran just fine suddenly didn't. The one-time repair aligned them with the newer expectations and the issues did not recur. I don't know for sure that this is what you experienced, of course.
* First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start.
Did you capture logs from your init system and representative OSDs? They would help understand what happened.
We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore.
It's good to have switched, though of course doing so in a planned fashion is always less stressful.
Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us.
Does your CRUSH map set optimal tunables? Or an older profile? # ceph osd crush show-tunables { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 1, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 } Older tunables can result in unequal data distribution. Similarly, are all of your CRUSH buckets straw2? # ceph osd crush dump | fgrep alg\" | sort | uniq -c 42 "alg": "straw2", If not ceph osd crush set-all-straw-buckets-to-straw2 That should help with uniformity, though note that it will cause data to move, and if you're using legacy OSD reweighs those values would need to be readjusted. What does ceph balancer status show? If you have legacy reweighs set to < 1.00 and pg-upmap balancing at the same time, you'll end up with outliers. When using pg-upmap balancing, one really has to reset all the legacy reweights. If the cluster is fairly full that may need to be done incrementally to minimize making outliers worse. Similarly, when using the upmap balancer, the CERN upmap-remapped tool can help avoid surprise full OSDs: https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/gracefully-expanding-your-ibm-storage-ceph Gracefully Expanding Your IBM Storage Ceph Cluster community.ibm.com
I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help).
Compaction can't happen until backfill/recovery is complete. At one point there was a bug when it also required that the numbers of total, up, and in OSDs were equal, i.e. all OSDs were up and in.
After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer).
Check your legacy reweights: # ceph osd tree | head ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -37 0 root staging -1 5577.10254 root default -34 465.31158 host cephab92 217 hdd 18.53969 osd.217 up 1.00000 1.00000 If you have any reweights that aren't 1.0000, that could be a factor. When using the upmap balancer, they all really need to be 1.0000.
*** Questions / Assumptions / Opinions ***
Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution
Filestore vs BlueStore is below RADOS, so it's not so bad. BlueStore OSDs are much less prone to memory ballooning but there's no special risk in running both that I've ever seen.
Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ?
Did you see any messages about legacy / per-pool stats? At a certain point, I don't recall when, a nifty new feature was added that required that BlueStore OSDs get a one-time repair, which could be done at startup, but which could take a while especially on spinners.
I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case:
root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’,
This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up?
I don't *think* scrubs are related to such cleanup, though when addressing large omaps, a scrub can be required for them to stop being *reported*.
(It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).)
Central config mostly means we don't have to inject any more. Much more convenient.
Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Wasn't the default for bluestore_min_alloc_size changed between N and P from 64k to 4k? That could also play a role when you had to rebuild your OSDs. And wasn't there also a Bluestore tombstone cleanup introduced? I don't know when exactly that was, but all these things could play a role, in addition to what the others wrote, especially the relatively small SSD OSDs could play a bigger role as Anthony already suggested. So it won't be easy to pinpoint the root cause, there are too many factors involved, unless you have captured logs and some additional terminal output during the rebuild like 'ceph osd df', or how big the PGs on the SSDs are and how exactly you rebuilt the OSDs. A lot of questions... Zitat von Anthony D'Atri <anthony.datri@gmail.com>:
From Hammer to the Pacific version is already a long journey.
I've seen some odd mon behavior from clusters initially installed on Hammer, but that surfaced in Jewel and is I doubt in play here.
I can't say that we upgraded a lot of clusters from N to P, but those that we upgraded didn't show any of these symptoms you describe. But we always did the Filestore to Bluestore conversion before the actual upgrade. In SUSE Enterprise Storage (which we also supported at that time) this was pointed out as a requirement. I just checked the ceph docs, I can't find such a statement (yet).
I *think* Filestore OSDs still work, but I've been peppering the docs with admonitions to convert for several releases now. I would expect them to have worked with Pacific. Did you update from the last Nautilus to the last Pacific?
All our pools are of the following type: replicated size 3 min_size 1 crush_rule 0 (or 1).
I would recommend to increase min_size to 2, otherwise you let Ceph lose two of three PGs before pausing IO, this can make recovery difficult. Reducing min_size to 1 should only be a temporary solution to preserve stalling client IO during recovery.
Absolutely.
. First, I need to explain the context of our CEPH
Ceph, please. Not CEPH. ;)
In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient.
Nice planning. For scratch clusters today I usually suggest a single 25-100 GE bonded public network. The dynamics when you started were different.
Each of the storage servers comes with HDDs whose size varies depending on the server generation
Good job keeping each DC's aggregate CRUSH weight nearly identical.
as well as SSDs whose size is more consistent but still varies (depending on price).
Enterprise SAS/SATA SSDs are starting to disappear from the market. The next time you buy servers, consider NVMe-only chassis. With careful procurement and by not having to pay for an HBA, they can be more affordable than you might think, and conserve precious RUs. And larger SSDs get you more capacity per chassis, so you can save on chassis, switch ports, etc.
100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB
Have you tried disabling their volatile write cache?
84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB
Just for others reading, very small SSDs can end up using a surprising fraction of their capacity for DB/WAL/other overhead, resulting in less usable capacity than one expects.
Regarding the CRUSHmap, and since at the time the CEPH cluster was launched, classes (ssd/hdd) did not exist and we wanted to be able to create pools on disk storage or flash storage, we created two trees:
Indeed, that was a common strategy. Device classes are way more convenient and work better with the upmap balancer. At some point you might consider converting with the crush tool reclassify feature, see: https://docs.ceph.com/en/pacific/rados/operations/crush-map-edits/
Everything was working fine until last August, when we scheduled an update from CEPH 14.x (Nautilus) to 16.X (Pacific) (and an update from Debian 10 to Debian 11, which was not a problem).
I'm not experienced with Debian as such, but Filestore OSDs being XFS filesystems are susceptible to XFS flern, which means both the kernel and filesystem utilities. I've seen [non-Ceph] issues with a large kernel version jump. In those situations a one-time xfs_repair addressed the problem. I don't recall the details, but ISTR that at a certain point XFS started paying closer attention to a certain filesystem structure than it used to, so older filesystems that previously ran just fine suddenly didn't. The one-time repair aligned them with the newer expectations and the issues did not recur. I don't know for sure that this is what you experienced, of course.
* First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start.
Did you capture logs from your init system and representative OSDs? They would help understand what happened.
We did not have this problem on our small test cluster, which obviously did not have the ‘same upgrade life’ as the production cluster. We therefore took the opportunity, DC by DC (since this is our ‘failure domain’), not only to update CEPH but also to recreate the OSDs in BlueStore.
It's good to have switched, though of course doing so in a planned fashion is always less stressful.
Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us.
Does your CRUSH map set optimal tunables? Or an older profile?
# ceph osd crush show-tunables { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 1, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }
Older tunables can result in unequal data distribution. Similarly, are all of your CRUSH buckets straw2?
# ceph osd crush dump | fgrep alg\" | sort | uniq -c 42 "alg": "straw2",
If not
ceph osd crush set-all-straw-buckets-to-straw2
That should help with uniformity, though note that it will cause data to move, and if you're using legacy OSD reweighs those values would need to be readjusted.
What does
ceph balancer status
show? If you have legacy reweighs set to < 1.00 and pg-upmap balancing at the same time, you'll end up with outliers. When using pg-upmap balancing, one really has to reset all the legacy reweights. If the cluster is fairly full that may need to be done incrementally to minimize making outliers worse.
Similarly, when using the upmap balancer, the CERN upmap-remapped tool can help avoid surprise full OSDs:
https://community.ibm.com/community/user/blogs/anthony-datri/2025/07/30/gracefully-expanding-your-ibm-storage-ceph Gracefully Expanding Your IBM Storage Ceph Cluster community.ibm.com
I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help).
Compaction can't happen until backfill/recovery is complete. At one point there was a bug when it also required that the numbers of total, up, and in OSDs were equal, i.e. all OSDs were up and in.
After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer).
Check your legacy reweights:
# ceph osd tree | head ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -37 0 root staging -1 5577.10254 root default -34 465.31158 host cephab92 217 hdd 18.53969 osd.217 up 1.00000 1.00000
If you have any reweights that aren't 1.0000, that could be a factor. When using the upmap balancer, they all really need to be 1.0000.
*** Questions / Assumptions / Opinions ***
Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution
Filestore vs BlueStore is below RADOS, so it's not so bad. BlueStore OSDs are much less prone to memory ballooning but there's no special risk in running both that I've ever seen.
Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ?
Did you see any messages about legacy / per-pool stats? At a certain point, I don't recall when, a nifty new feature was added that required that BlueStore OSDs get a one-time repair, which could be done at startup, but which could take a while especially on spinners.
I also read on the Internet (somewhere...) that in any case, when the cluster is not ‘healthy’, scrubs are suspended by default. Indeed, in our case:
root@cephstore16:~# ceph daemon osd.11636 config show | grep ‘osd_scrub_during_recovery’ ‘osd_scrub_during_recovery’: ‘false’,
This could explain why, during the three days of recovery, no cleaning is performed and if bluestore does not perform maintenance, it fills up?
I don't *think* scrubs are related to such cleanup, though when addressing large omaps, a scrub can be required for them to stop being *reported*.
(It would be possible to temporarily change this behaviour via: ceph tell “osd.*” injectargs --osd-scrub-during-recovery=1 (to be tested).)
Central config mostly means we don't have to inject any more. Much more convenient.
Do you have any suggestions for things to check? Although we have experience with FileStore, we have not yet had time to gain experience with BlueStore.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 14 sept. 2025 à 10:15, Eugen Block <eblock@nde.ag> a écrit :
Wasn't the default for bluestore_min_alloc_size changed between N and P from 64k to 4k? That could also play a role when you had to rebuild your OSDs. And wasn't there also a Bluestore tombstone cleanup introduced? I don't know when exactly that was, but all these things could play a role, in addition to what the others wrote, especially the relatively small SSD OSDs could play a bigger role as Anthony already suggested. So it won't be easy to pinpoint the root cause, there are too many factors involved, unless you have captured logs and some additional terminal output during the rebuild like 'ceph osd df', or how big the PGs on the SSDs are and how exactly you rebuilt the OSDs. A lot of questions…
Yes indeed. Good questions…. Don’t have the answer. But it gives me maybe some clues to dig in. Olivier
Hello, Regarding our Ceph issue, we have a new theory that has emerged from our findings. This morning, we had an OSD crash because the underlying disk had bad blocks. This OSD, still on a machine that needs to be updated and converted, saw its XFS FS crash due to the bad blocks. In itself, this is nothing unusual (with filestore). What we did observe, however, is that during the recovery and rebalancing phase—while the OSD was down, and then during ‘its’ recovery—the MON DB ballooned (~ 3 GB -> 30 GB) and the ‘SSD’ OSDs saw their occupancy increase 0.01% by 0.01% (we took about 2% over ~ 4 hours). Once the recovery was complete and the cluster status returned to ‘Healthy’, the MON DB returned to its normal size and the ‘SSD’ OSDs saw their occupancy return to a normal level, i.e. the level before the incident. So our assumption now, which is more rational, is that during recovery, the cluster keeps many more (if not all?) PGmaps, which explains why the MON DB swells and why the OSDs probably also keep (would keep) as many copies as when the cluster is in an OK state. The questions that now arise are: why does an event as simple as an OSD down cause such a significant swelling of the MON DB (we did not have this before the upgrade and conversion)? And why does the OSD space usage reach such a point that it can become saturated? In any case, based on our previous assumption, the connection point is likely that scrubs are suspended when the cluster is not ‘healthy’, hence our observation/assumption that restarting scrubs could have an impact, which is probably not the case. And in relation to our upgrade/conversion plan, doing it server by server seems safer. Stay tuned... Olivier
Le 13 sept. 2025 à 14:17, Anthony D'Atri <anthony.datri@gmail.com> a écrit :
(..)
I can't say that we upgraded a lot of clusters from N to P, but those that we upgraded didn't show any of these symptoms you describe. But we always did the Filestore to Bluestore conversion before the actual upgrade. In SUSE Enterprise Storage (which we also supported at that time) this was pointed out as a requirement. I just checked the ceph docs, I can't find such a statement (yet).
I *think* Filestore OSDs still work, but I've been peppering the docs with admonitions to convert for several releases now. I would expect them to have worked with Pacific. Did you update from the last Nautilus to the last Pacific?
Yes. And yes it was supposed to work. And in fact it works on our test cluster but this one has not the same history as the prod one and it’s there only to test some crush rules,… before applying to prod cluster.
In terms of hardware, we have three monitors (cephmon) and 30 storage servers (cephstore) spread across three datacenters. These servers are connected to the network via an aggregate (LACP) of two 10 Gbps fibre connections, through which two VLANs pass, one for the CEPH frontend network and one for the CEPH backend network. In doing so, we have always given ourselves the option of separating the frontend and backend into dedicated aggregates if the bandwidth becomes insufficient.
Nice planning. For scratch clusters today I usually suggest a single 25-100 GE bonded public network. The dynamics when you started were different.
Yes. But for a network availability of ports point of view, I thing the next step will be probably splitting the front and the back network.
100 hdd 10.90999 TB 48 hdd 11.00000 TB 48 hdd 14.54999 TB 24 hdd 15.00000 TB 9 hdd 5.45999 TB 108 hdd 9.09999 TB
Have you tried disabling their volatile write cache?
Yes they are.
84 ssd 0.89400 TB 198 ssd 0.89424 TB 18 ssd 0.93599 TB 32 ssd 1.45999 TB 16 ssd 1.50000 TB 48 ssd 1.75000 TB 24 ssd 1.79999 TB
Just for others reading, very small SSDs can end up using a surprising fraction of their capacity for DB/WAL/other overhead, resulting in less usable capacity than one expects.
DB is always offloaded to a RAID1 NVMe storage. (..)
* First problem: We were forced to switch from FileStore to BlueStore in an emergency and unscheduled manner because after upgrading the CEPH packages on the first storage server, the FileStore OSDs would no longer start.
Did you capture logs from your init system and representative OSDs? They would help understand what happened.
Unfortunately not. We were not expecting the problems when we started the upgrade… :-( (..)
Here we see that our SSD-type OSDs fill up at a rate of ~ 2% every 3 hours (the phenomenon is also observed on HDD-type OSDs, but as we have a large capacity, it is less critical). Manual (re)weight changes only provided a temporary solution and, despite all our attempts (OSD restart, etc.), we reached the critical full_ratio threshold, which is 0.97 for us.
Does your CRUSH map set optimal tunables? Or an older profile?
# ceph osd crush show-tunables { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 1, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }
Seems the same as yours { "choose_local_tries": 0, "choose_local_fallback_tries": 0, "choose_total_tries": 50, "chooseleaf_descend_once": 1, "chooseleaf_vary_r": 1, "chooseleaf_stable": 1, "straw_calc_version": 1, "allowed_bucket_algs": 54, "profile": "jewel", "optimal_tunables": 1, "legacy_tunables": 0, "minimum_required_version": "jewel", "require_feature_tunables": 1, "require_feature_tunables2": 1, "has_v2_rules": 0, "require_feature_tunables3": 1, "has_v3_rules": 0, "has_v4_buckets": 1, "require_feature_tunables5": 1, "has_v5_rules": 0 }
Older tunables can result in unequal data distribution. Similarly, are all of your CRUSH buckets straw2?
# ceph osd crush dump | fgrep alg\" | sort | uniq -c 42 "alg": "straw2",
Yup 306 "alg": "straw2",
If not
ceph osd crush set-all-straw-buckets-to-straw2
That should help with uniformity, though note that it will cause data to move, and if you're using legacy OSD reweighs those values would need to be readjusted.
What does
ceph balancer status
show? If you have legacy reweighs set to < 1.00 and pg-upmap balancing at the same time, you'll end up with outliers. When using pg-upmap balancing, one really has to reset all the legacy reweights. If the cluster is fairly full that may need to be done incrementally to minimize making outliers worse.
{ "active": true, "last_optimize_duration": "0:00:01.032868", "last_optimize_started": "Sun Sep 14 11:58:51 2025", "mode": "upmap", "no_optimization_needed": true, "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] }
I'll leave you to imagine the effect on the virtual machines and the services provided to our users. We also had very strong growth in the size of the MONitor databases (~3 GB -> 100 GB) (compaction did not really help).
Compaction can't happen until backfill/recovery is complete. At one point there was a bug when it also required that the numbers of total, up, and in OSDs were equal, i.e. all OSDs were up and in.
Good to know
After this second total recovery of the CEPH cluster and the restart of the virtualisation environment, we still have the third DC (10 cephstore) to update from CEPH 14 to 16, and our ‘SSD’ OSDs are filling up again until the automatic activation of scrubs/deep-scrubs at 7 p.m. Since then, progress has stopped, the use of the various OSDs is stable and more or less evenly distributed (via active upmap balancer).
Check your legacy reweights:
# ceph osd tree | head ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -37 0 root staging -1 5577.10254 root default -34 465.31158 host cephab92 217 hdd 18.53969 osd.217 up 1.00000 1.00000
If you have any reweights that aren't 1.0000, that could be a factor. When using the upmap balancer, they all really need to be 1.0000.
Make sense
*** Questions / Assumptions / Opinions ***
Have you ever encountered a similar phenomenon? We agree that having different versions of OSDs coexisting is not a good solution
Filestore vs BlueStore is below RADOS, so it's not so bad. BlueStore OSDs are much less prone to memory ballooning but there's no special risk in running both that I've ever seen.
OK
Our current hypothesis, following the restoration of stability and the fact that we have never had this problem with OSDs in FileStore, is that there is some kind of ‘housekeeping’ of BlueStore OSDs via scrubs. Does that make sense? Any clues ? ideas ?
Did you see any messages about legacy / per-pool stats? At a certain point, I don't recall when, a nifty new feature was added that required that BlueStore OSDs get a one-time repair, which could be done at startup, but which could take a while especially on spinners.
No msg but the time for the 1st start can be really long…. Guess it’s what you describe, the one-time repair / check ? Thx for your answers. Olivier
Le 13 sept. 2025 à 08:45, Eugen Block <eblock@nde.ag> a écrit :
Hi,
I can't say that we upgraded a lot of clusters from N to P, but those that we upgraded didn't show any of these symptoms you describe. But we always did the Filestore to Bluestore conversion before the actual upgrade. In SUSE Enterprise Storage (which we also supported at that time) this was pointed out as a requirement. I just checked the ceph docs, I can't find such a statement (yet).
Hi, Thanks for your answer. Maybe we should have to the switch from FS to BS before but for me, Pacific was the last version of Ceph supporting FS. So we were supposed to have “time”. Olivier
participants (5)
-
Anthony D'Atri
-
Eugen Block
-
Joachim Kraftmayer
-
Michel Jouvin
-
Olivier Delcourt