How to make HEALTH_ERR quickly and pain-free
I have hell of the question: how to make HEALTH_ERR status for a cluster without consequences? I'm working on CI tests and I need to check if our reaction to HEALTH_ERR is good. For this I need to take an empty cluster with an empty pool and do something. Preferably quick and reversible. For HEALTH_WARN the best thing I found is to change pool size to 1, it raises "1 pool(s) have no replicas configured" warning almost instantly and it can be reverted very quickly for empty pool. But HEALTH_ERR is a bit more tricky. Any ideas?
Hi,
For HEALTH_WARN the best thing I found is to change pool size to 1, it raises "1 pool(s) have no replicas configured" warning almost instantly and it can be reverted very quickly for empty pool.
any osd flag (noout, nodeep-scrub etc.) cause health warnings. ;-)
But HEALTH_ERR is a bit more tricky. Any ideas?
I think if you set a very low quota for a pool (e.g. 1000 bytes or so) and fill it up it should create a HEALTH_ERR status, IIRC. Zitat von George Shuklin <george.shuklin@gmail.com>:
I have hell of the question: how to make HEALTH_ERR status for a cluster without consequences?
I'm working on CI tests and I need to check if our reaction to HEALTH_ERR is good. For this I need to take an empty cluster with an empty pool and do something. Preferably quick and reversible.
For HEALTH_WARN the best thing I found is to change pool size to 1, it raises "1 pool(s) have no replicas configured" warning almost instantly and it can be reverted very quickly for empty pool.
But HEALTH_ERR is a bit more tricky. Any ideas? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
But HEALTH_ERR is a bit more tricky. Any ideas?
I think if you set a very low quota for a pool (e.g. 1000 bytes or so) and fill it up it should create a HEALTH_ERR status, IIRC. Cool idea. Unfortunately, even with 1 byte quota (and some data in the
On 21/01/2021 13:02, Eugen Block wrote: pool), it's HEALTH_WARN, 1 pool(s) full
Oh really, I thought it would be an error. My bad. There was an osd flag "full" which is not usable anymore, I never used it so I just tried it with full OSD which should lead to an error (and it does): host:~ # ceph -s cluster: id: 8f279f36-811c-3270-9f9d-58335b1bb9c0 health: HEALTH_ERR 1 full osd(s) 22 pool(s) full A created a pool with 1 pg and size 1, created an rbd image in that pool and filled that up until the respective OSD was full. I have a virtual lab cluster with 20GB OSDs so it didn't take that long. If you try this, make sure to disable pg-autoscaler on that pool or otherwise it will increase pg-num. Does that help? Regards, Eugen Zitat von George Shuklin <george.shuklin@gmail.com>:
But HEALTH_ERR is a bit more tricky. Any ideas?
I think if you set a very low quota for a pool (e.g. 1000 bytes or so) and fill it up it should create a HEALTH_ERR status, IIRC. Cool idea. Unfortunately, even with 1 byte quota (and some data in
On 21/01/2021 13:02, Eugen Block wrote: the pool), it's HEALTH_WARN, 1 pool(s) full _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 21/01/2021 12:57, George Shuklin wrote:
I have hell of the question: how to make HEALTH_ERR status for a cluster without consequences?
I'm working on CI tests and I need to check if our reaction to HEALTH_ERR is good. For this I need to take an empty cluster with an empty pool and do something. Preferably quick and reversible.
For HEALTH_WARN the best thing I found is to change pool size to 1, it raises "1 pool(s) have no replicas configured" warning almost instantly and it can be reverted very quickly for empty pool.
But HEALTH_ERR is a bit more tricky. Any ideas?
I found the way: ceph osd set-full-ratio 0.0 instantly causing health: HEALTH_ERR full ratio(s) out of order even on empty cluster. Problem solved.
Oh that's better, I had to recreate my OSD because it didn't want to start anymore :-D Zitat von George Shuklin <george.shuklin@gmail.com>:
On 21/01/2021 12:57, George Shuklin wrote:
I have hell of the question: how to make HEALTH_ERR status for a cluster without consequences?
I'm working on CI tests and I need to check if our reaction to HEALTH_ERR is good. For this I need to take an empty cluster with an empty pool and do something. Preferably quick and reversible.
For HEALTH_WARN the best thing I found is to change pool size to 1, it raises "1 pool(s) have no replicas configured" warning almost instantly and it can be reverted very quickly for empty pool.
But HEALTH_ERR is a bit more tricky. Any ideas?
I found the way:
ceph osd set-full-ratio 0.0
instantly causing
health: HEALTH_ERR full ratio(s) out of order
even on empty cluster. Problem solved. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Eugen Block
-
George Shuklin