BlueFS spillover warning gone after upgrade to Quincy
Hello everyone I have a Ceph installation where some of the OSDs were misconfigured to use 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS *spillover* detected"). I recently upgraded to quincy using cephadm (17.2.5) the spillover warning vanished. This is despite bluestore_warn_on_bluefs_spillover still being set to true. Is there a way to investigate the current state of the DB to see if spillover is, indeed, still happening? Thank you, Peter
Hi, I usually look for this: [ceph: root@storage01 /]# ceph daemon osd.0 perf dump bluefs | grep -E "db_|slow_" "db_total_bytes": 21470642176, "db_used_bytes": 179699712, "slow_total_bytes": 0, "slow_used_bytes": 0, If you have spillover I would expect the "slow_bytes" values to be > 0. Is it possible that the OSDs were compacted during/after the upgrade so the spillover would have been corrected (temporarily)? Do you know how much spillover you had before? And how big was the db when you had the warnings? Regards, Eugen Zitat von Peter van Heusden <pvh@sanbi.ac.za>:
Hello everyone
I have a Ceph installation where some of the OSDs were misconfigured to use 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS *spillover* detected"). I recently upgraded to quincy using cephadm (17.2.5) the spillover warning vanished. This is despite bluestore_warn_on_bluefs_spillover still being set to true.
Is there a way to investigate the current state of the DB to see if spillover is, indeed, still happening?
Thank you, Peter _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks. The command definitely shows "slow_bytes": "db_total_bytes": 1073733632, "db_used_bytes": 240123904, "slow_total_bytes": 4000681103360, "slow_used_bytes": 8355381248, So I am not sure why the warnings are no longer appearing. Peter On Thu, 12 Jan 2023 at 17:41, Eugen Block <eblock@nde.ag> wrote:
Hi,
I usually look for this:
[ceph: root@storage01 /]# ceph daemon osd.0 perf dump bluefs | grep -E "db_|slow_" "db_total_bytes": 21470642176, "db_used_bytes": 179699712, "slow_total_bytes": 0, "slow_used_bytes": 0,
If you have spillover I would expect the "slow_bytes" values to be > 0. Is it possible that the OSDs were compacted during/after the upgrade so the spillover would have been corrected (temporarily)? Do you know how much spillover you had before? And how big was the db when you had the warnings?
Regards, Eugen
Zitat von Peter van Heusden <pvh@sanbi.ac.za>:
Hello everyone
I have a Ceph installation where some of the OSDs were misconfigured to use 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS *spillover* detected"). I recently upgraded to quincy using cephadm (17.2.5) the spillover warning vanished. This is despite bluestore_warn_on_bluefs_spillover still being set to true.
Is there a way to investigate the current state of the DB to see if spillover is, indeed, still happening?
Thank you, Peter _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
If you have prometheus enabled, the metrics should be in there I think? Thanks, Kevin ________________________________________ From: Peter van Heusden <pvh@sanbi.ac.za> Sent: Thursday, January 12, 2023 6:12 AM To: ceph-users@ceph.io Subject: [ceph-users] BlueFS spillover warning gone after upgrade to Quincy Check twice before you click! This email originated from outside PNNL. Hello everyone I have a Ceph installation where some of the OSDs were misconfigured to use 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS *spillover* detected"). I recently upgraded to quincy using cephadm (17.2.5) the spillover warning vanished. This is despite bluestore_warn_on_bluefs_spillover still being set to true. Is there a way to investigate the current state of the DB to see if spillover is, indeed, still happening? Thank you, Peter _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Peter, On Thursday, January 12th, 2023 at 15:12, Peter van Heusden <pvh@sanbi.ac.za> wrote:
I have a Ceph installation where some of the OSDs were misconfigured to use 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS spillover detected"). I recently upgraded to quincy using cephadm (17.2.5) the spillover warning vanished. This is despite bluestore_warn_on_bluefs_spillover still being set to true.
I noticed this on Pacific as well, and I think it's due to this commit: https://github.com/ceph/ceph/commit/d17cd6604b4031ca997deddc5440248aff451269. It removes the logic that would normally update the spillover health check, so it never triggers anymore. As others mentioned, you can get the relevant metrics from Prometheus and setup alerts there instead. But it does make me wonder how many people might have spillover in their clusters and not even realize it, since there's no warning by default. Cheers, -- Ben
Hi Benoit and Peter, looks like your findings are valid and spillover alert is broken for now. I've just created https://tracker.ceph.com/issues/58440 to track this. Thanks, Igor On 1/13/2023 9:54 AM, Benoît Knecht wrote:
Hi Peter,
On Thursday, January 12th, 2023 at 15:12, Peter van Heusden<pvh@sanbi.ac.za> wrote:
I have a Ceph installation where some of the OSDs were misconfigured to use 1GB SSD partitions for rocksdb. This caused a spillover ("BlueFS spillover detected"). I recently upgraded to quincy using cephadm (17.2.5) the spillover warning vanished. This is despite bluestore_warn_on_bluefs_spillover still being set to true. I noticed this on Pacific as well, and I think it's due to this commit: https://github.com/ceph/ceph/commit/d17cd6604b4031ca997deddc5440248aff451269. It removes the logic that would normally update the spillover health check, so it never triggers anymore.
As others mentioned, you can get the relevant metrics from Prometheus and setup alerts there instead. But it does make me wonder how many people might have spillover in their clusters and not even realize it, since there's no warning by default.
Cheers,
-- Igor Fedotov Ceph Lead Developer -- croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web <https://croit.io/> | LinkedIn <http://linkedin.com/company/croit> | Youtube <https://www.youtube.com/channel/UCIJJSKVdcSLGLBtwSFx_epw> | Twitter <https://twitter.com/croit_io> Meet us at the SC22 Conference! Learn more <https://croit.io/croit-sc22> Technology Fast50 Award Winner by Deloitte <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>! <https://www2.deloitte.com/de/de/pages/technology-media-and-telecommunications/articles/fast-50-2022-germany-winners.html>
participants (5)
-
Benoît Knecht
-
Eugen Block
-
Fox, Kevin M
-
Igor Fedotov
-
Peter van Heusden