1 Dec
2023
1 Dec
'23
2:04 p.m.
Hi Stefan,
I am running replicate x3 with a failure domain as host and setting
min_size pool is 1. Because my cluster s3 traffic real time and can't stop
or block IO, the data may be lost but IO alway available. I hope my cluster
can run with two nodes unavailable.
After that two nodes is down at the same time, and then nodes up, client IO
and recover running in the same time, and some disk warning is slowops,
what is the problem, may be my disk is overload, but the disk utilization
only 60 -80%
Thanks Stefan
Vào Th 6, 1 thg 12, 2023 vào lúc 16:40 Stefan Kooman <stefan(a)bit.nl> đã
viết:
On 01-12-2023 08:45, VÔ VI wrote:
Hi community,
My cluster running with 10 nodes and 2 nodes goes down, sometimes the log
shows the slow ops, what is the root cause?
My osd is HDD and block.db and wal is 500GB SSD per osd.
Health check update: 13 slow ops, oldest one blocked for 167 sec, osd.10
has slow ops (SLOW_OPS)
Most likely you have a crush rule that spreads objects over hosts as a
failure domain. For size=3, min_size=2 (default for replicated pools)
you might end up in a situation where two of the nodes that are offline
have PGs where min_size=2 requirement is not fulfilled, and will hence
by inactive and slow ops will occur.
When host is your failure domain, you should not reboot more than one at
the same time. If the hosts are somehow organized (different racks,
datacenters) you could make a higher level bucket and put your hosts
there. And create a crush rule using that bucket type as failure domain,
and have your pools use that.
Gr. Stefan