6 Oct
2020
6 Oct
'20
1:10 p.m.
Hi everybody,
Our need is to do VM failover using an image disk over RBD to avoid data loss.We want to
limit the downtime as much as
possible.
We have: - Two hypervisors with a Ceph Monitor and a Ceph OSD. - A third machine with a
Ceph Monitor and a Ceph
Manager.
VM are running over qemu.The VM disks are on a "replicated" rbd pool formed by
the two OSDs.Ceph version:
NautilusDistribution: Yocto Zeus
The following test is performed: we electrically turn off one hypervisor (and therefore a
Ceph Monitor and a Ceph OSD),
which causes its VMs to switch to the second hypervisor.
My main issue is that the mount time of a partition in rw is very slow in the case of a
failover (after the loss of an
OSD its monitor).
With failover we can write on the device after ~25s:[ 25.609074] EXT4-fs (vda3): mounted
filesystem with ordered data
mode. Opts: (null)
In normal boot we can write on the device after ~4s:[ 3.087412] EXT4-fs (vda3): mounted
filesystem with ordered data
mode. Opts: (null)
I wasn't able to reduce this time by tweaking Ceph settings. I am wondering if someone
could help me on that.
Here is our configuration.
ceph.conf[global] fsid = fa7a17d1-5351-459e-bf0e-07e7edc9a625 mon initial members =
hypervisor1,hypervisor2,observer mon host =
192.168.217.131,192.168.217.132,192.168.217.133 public network =
192.168.217.0/24 auth cluster required = cephx auth service required = cephx auth
client required =
cephx osd journal size = 1024 osd pool default size = 2 osd pool default min size
= 1 osd crush chooseleaf
type = 1 mon osd adjust heartbeat grace = false mon osd min down reporters =
1[mon.hypervisor1] host =
hypervisor1 mon addr = 192.168.217.131:6789[mon.hypervisor2] host = hypervisor2
mon addr =
192.168.217.132:6789[mon.observer] host = observer mon addr =
192.168.217.133:6789[osd.0] host =
hypervisor1 public_addr = 192.168.217.131 cluster_addr = 192.168.217.131[osd.1]
host =
hypervisor2 public_addr = 192.168.217.132 cluster_addr = 192.168.217.13
# ceph config dump WHO MASK LEVEL OPTION VALUE RO
global advanced
mon_osd_adjust_down_out_interval false global advanced
mon_osd_adjust_heartbeat_grace false global advanced
mon_osd_down_out_interval 5 global advanced
mon_osd_report_timeout 4 global advanced
osd_beacon_report_interval 1 global advanced
osd_heartbeat_grace 2 global advanced
osd_heartbeat_interval 1 global advanced
osd_mon_ack_timeout 1.000000 global advanced
osd_mon_heartbeat_interval 2 global advanced osd_mon_report_interval
3
Thanks
6 Oct
6 Oct
1:34 p.m.
I think I do not understand you completely. How long does a live
migration take? If I do virsh migrate with vm's on librbd it is a few
seconds. I guess this is mainly caused by copying the ram to the other
host.
Any time more this takes in case of a host failure, is related to time
out settings, failure detection and locks being released etc.
-----Original Message-----
From: Mathieu Dupré [mailto:mathieu.dupre@savoirfairelinux.com]
Sent: dinsdag 6 oktober 2020 9:40
To: ceph-users(a)ceph.io
Cc: Bail, Eloi
Subject: [ceph-users] Write access delay after OSD & Mon lost
Hi everybody,
Our need is to do VM failover using an image disk over RBD to avoid data
loss.We want to limit the downtime as much as possible.
We have: - Two hypervisors with a Ceph Monitor and a Ceph OSD. - A third
machine with a Ceph Monitor and a Ceph Manager.
VM are running over qemu.The VM disks are on a "replicated" rbd pool
formed by the two OSDs.Ceph version:
NautilusDistribution: Yocto Zeus
The following test is performed: we electrically turn off one hypervisor
(and therefore a Ceph Monitor and a Ceph OSD), which causes its VMs to
switch to the second hypervisor.
My main issue is that the mount time of a partition in rw is very slow
in the case of a failover (after the loss of an OSD its monitor).
With failover we can write on the device after ~25s:[ 25.609074]
EXT4-fs (vda3): mounted filesystem with ordered data
mode. Opts: (null)
In normal boot we can write on the device after ~4s:[ 3.087412]
EXT4-fs (vda3): mounted filesystem with ordered data
mode. Opts: (null)
I wasn't able to reduce this time by tweaking Ceph settings. I am
wondering if someone could help me on that.
Here is our configuration.
ceph.conf[global] fsid = fa7a17d1-5351-459e-bf0e-07e7edc9a625 mon
initial members =
hypervisor1,hypervisor2,observer mon host =
192.168.217.131,192.168.217.132,192.168.217.133 public network =
192.168.217.0/24 auth cluster required = cephx auth service
required = cephx auth client required =
cephx osd journal size = 1024 osd pool default size = 2 osd
pool default min size = 1 osd crush chooseleaf
type = 1 mon osd adjust heartbeat grace = false mon osd min down
reporters = 1[mon.hypervisor1] host =
hypervisor1 mon addr = 192.168.217.131:6789[mon.hypervisor2] host
= hypervisor2 mon addr =
192.168.217.132:6789[mon.observer] host = observer mon addr =
192.168.217.133:6789[osd.0] host =
hypervisor1 public_addr = 192.168.217.131 cluster_addr =
192.168.217.131[osd.1] host =
hypervisor2 public_addr = 192.168.217.132 cluster_addr =
192.168.217.13
# ceph config dump WHO MASK LEVEL OPTION
VALUE RO global advanced
mon_osd_adjust_down_out_interval false global advanced
mon_osd_adjust_heartbeat_grace false global advanced
mon_osd_down_out_interval 5 global advanced
mon_osd_report_timeout 4 global advanced
osd_beacon_report_interval 1 global advanced
osd_heartbeat_grace 2 global advanced
osd_heartbeat_interval 1 global advanced
osd_mon_ack_timeout 1.000000 global advanced
osd_mon_heartbeat_interval 2 global advanced
osd_mon_report_interval 3
Thanks
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
3:20 p.m.
IF an OSD is lost, it will be detected after
osd heartbeat grace = 20 +
osd heartbeat interval = 5
ie 25 sec by default, which is what you see. During this time client io
will block, after this the OSD is flagged as down and a new OSD map is
issued which the client will use to re-direct the blocked io.
You can lower it, but 25 sec is a reasonable failover time, the risk of
lowering it is that the down detection can be over sensitive and gets
triggered during heavy load which can lead to cluster instability.
Everyone would like to have quicker failover, but the default values
were set for a reason.
Cheers /Maged
On 06/10/2020 09:40, Mathieu Dupré wrote:
Hi everybody,
Our need is to do VM failover using an image disk over RBD to avoid data loss.We want to
limit the downtime as much as
possible.
We have: - Two hypervisors with a Ceph Monitor and a Ceph OSD. - A third machine with a
Ceph Monitor and a Ceph
Manager.
VM are running over qemu.The VM disks are on a "replicated" rbd pool formed by
the two OSDs.Ceph version:
NautilusDistribution: Yocto Zeus
The following test is performed: we electrically turn off one hypervisor (and therefore a
Ceph Monitor and a Ceph OSD),
which causes its VMs to switch to the second hypervisor.
My main issue is that the mount time of a partition in rw is very slow in the case of a
failover (after the loss of an
OSD its monitor).
With failover we can write on the device after ~25s:[ 25.609074] EXT4-fs (vda3):
mounted filesystem with ordered data
mode. Opts: (null)
In normal boot we can write on the device after ~4s:[ 3.087412] EXT4-fs (vda3):
mounted filesystem with ordered data
mode. Opts: (null)
I wasn't able to reduce this time by tweaking Ceph settings. I am wondering if
someone could help me on that.
Here is our configuration.
ceph.conf[global] fsid = fa7a17d1-5351-459e-bf0e-07e7edc9a625 mon initial members
=
hypervisor1,hypervisor2,observer mon host =
192.168.217.131,192.168.217.132,192.168.217.133 public network =
192.168.217.0/24 auth cluster required = cephx auth service required = cephx
auth client required =
cephx osd journal size = 1024 osd pool default size = 2 osd pool default min
size = 1 osd crush chooseleaf
type = 1 mon osd adjust heartbeat grace = false mon osd min down reporters =
1[mon.hypervisor1] host =
hypervisor1 mon addr = 192.168.217.131:6789[mon.hypervisor2] host = hypervisor2
mon addr =
192.168.217.132:6789[mon.observer] host = observer mon addr =
192.168.217.133:6789[osd.0] host =
hypervisor1 public_addr = 192.168.217.131 cluster_addr = 192.168.217.131[osd.1]
host =
hypervisor2 public_addr = 192.168.217.132 cluster_addr = 192.168.217.13
# ceph config dump WHO MASK LEVEL OPTION VALUE RO
global advanced
mon_osd_adjust_down_out_interval false global advanced
mon_osd_adjust_heartbeat_grace false global advanced
mon_osd_down_out_interval 5 global advanced
mon_osd_report_timeout 4 global advanced
osd_beacon_report_interval 1 global advanced
osd_heartbeat_grace 2 global advanced
osd_heartbeat_interval 1 global advanced
osd_mon_ack_timeout 1.000000 global advanced
osd_mon_heartbeat_interval 2 global advanced osd_mon_report_interval
3
Thanks
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1306
days inactive
1306
days old
2 comments
3 participants
participants (3)
-
Maged Mokhtar -
Marc Roos -
Mathieu Dupré