16 May
2020
16 May
'20
4:11 p.m.
Hi,
I am using ceph Nautilus cluster with below configuration.
3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr are running
in shared mode.
The client mounted through ceph kernel client.
I was trying to emulate a node failure when a write and read were going on
(replica2) pool.
I was expecting read and write continue after a small pause due to a Node
failure but it halts and never resumes until the failed node is up.
I remember I tested the same scenario before in ceph mimic where it
continued IO after a small pause.
regards
Amudhan P
16 May
16 May
7:59 p.m.
What’s your pool configuration wrt min_size and crush rules?
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
Hi,
I am using ceph Nautilus cluster with below configuration.
3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr are running
in shared mode.
The client mounted through ceph kernel client.
I was trying to emulate a node failure when a write and read were going on
(replica2) pool.
I was expecting read and write continue after a small pause due to a Node
failure but it halts and never resumes until the failed node is up.
I remember I tested the same scenario before in ceph mimic where it
continued IO after a small pause.
regards
Amudhan P
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
17 May
17 May
4:43 p.m.
Hi,
Crush rule is "replicated" and min_size 2 actually. I am trying to test
multiple volume configs in a single filesystem
using file layout.
I have created metadata pool with rep 3 (min_size2 and replicated crush
rule) and data pool with rep 3 (min_size2 and replicated crush rule). and
also I have created multiple (replica 2, ec2-1 & ec4-2) pools and added to
the filesystem.
Using file layout I have set different data pool to a different folders. so
I can test different configs in the same filesystem. all data pools
min_size set to handle single node failure.
Single node failure is handled properly when only having metadata pool and
one data pool (rep3).
After adding additional data pool to fs, single node failure scenario is
not working.
regards
Amudhan P
On Sun, May 17, 2020 at 1:29 AM Eugen Block <eblock(a)nde.ag> wrote:
What’s your pool configuration wrt min_size and crush
rules?
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
Hi,
I am using ceph Nautilus cluster with below configuration.
3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr are
running
in shared mode.
The client mounted through ceph kernel client.
I was trying to emulate a node failure when a write and read were going
on
(replica2) pool.
I was expecting read and write continue after a small pause due to a Node
failure but it halts and never resumes until the failed node is up.
I remember I tested the same scenario before in ceph mimic where it
continued IO after a small pause.
regards
Amudhan P
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
18 May
18 May
7:04 a.m.
If your pool has a min_size 2 and size 2 (always a bad idea) it will
pause IO in case of a failure until the recovery has finished. So the
described behaviour is expected.
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
Hi,
Crush rule is "replicated" and min_size 2 actually. I am trying to test
multiple volume configs in a single filesystem
using file layout.
I have created metadata pool with rep 3 (min_size2 and replicated crush
rule) and data pool with rep 3 (min_size2 and replicated crush rule). and
also I have created multiple (replica 2, ec2-1 & ec4-2) pools and added to
the filesystem.
Using file layout I have set different data pool to a different folders. so
I can test different configs in the same filesystem. all data pools
min_size set to handle single node failure.
Single node failure is handled properly when only having metadata pool and
one data pool (rep3).
After adding additional data pool to fs, single node failure scenario is
not working.
regards
Amudhan P
On Sun, May 17, 2020 at 1:29 AM Eugen Block <eblock(a)nde.ag> wrote:
> What’s your pool configuration wrt min_size and crush rules?
>
>
> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
>
> > Hi,
> >
> > I am using ceph Nautilus cluster with below configuration.
> >
> > 3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr are
> running
> > in shared mode.
> >
> > The client mounted through ceph kernel client.
> >
> > I was trying to emulate a node failure when a write and read were going
> on
> > (replica2) pool.
> >
> > I was expecting read and write continue after a small pause due to a Node
> > failure but it halts and never resumes until the failed node is up.
> >
> > I remember I tested the same scenario before in ceph mimic where it
> > continued IO after a small pause.
> >
> > regards
> > Amudhan P
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
>
>
19 May
19 May
6:02 a.m.
Behaviour is same even after setting min_size 2.
On Mon 18 May, 2020, 12:34 PM Eugen Block, <eblock(a)nde.ag> wrote:
If your pool has a min_size 2 and size 2 (always a bad
idea) it will
pause IO in case of a failure until the recovery has finished. So the
described behaviour is expected.
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
Hi,
Crush rule is "replicated" and min_size 2 actually. I am trying to test
multiple volume configs in a single filesystem
using file layout.
I have created metadata pool with rep 3 (min_size2 and replicated crush
rule) and data pool with rep 3 (min_size2 and replicated crush rule).
and
also I have created multiple (replica 2, ec2-1
& ec4-2) pools and added
to
the filesystem.
Using file layout I have set different data pool to a different folders.
so
I can test different configs in the same
filesystem. all data pools
min_size set to handle single node failure.
Single node failure is handled properly when only having metadata pool
and
one data pool (rep3).
After adding additional data pool to fs, single node failure scenario is
not working.
regards
Amudhan P
On Sun, May 17, 2020 at 1:29 AM Eugen Block <eblock(a)nde.ag> wrote:
> What’s your pool configuration wrt min_size and crush rules?
>
>
> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
>
> > Hi,
> >
> > I am using ceph Nautilus cluster with below configuration.
> >
> > 3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr are
> running
> > in shared mode.
> >
> > The client mounted through ceph kernel client.
> >
> > I was trying to emulate a node failure when a write and read were
going
> on
> > (replica2) pool.
> >
> > I was expecting read and write continue after a small pause due to a
Node
> > failure but it halts and never resumes
until the failed node is up.
> >
> > I remember I tested the same scenario before in ceph mimic where it
> > continued IO after a small pause.
> >
> > regards
> > Amudhan P
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
>
>
>
6:46 a.m.
Was that a typo and you mean you changed min_size to 1? I/O paus with
min_size 1 and size 2 is unexpected, can you share more details like
your crushmap and your osd tree?
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
Behaviour is same even after setting min_size 2.
On Mon 18 May, 2020, 12:34 PM Eugen Block, <eblock(a)nde.ag> wrote:
> If your pool has a min_size 2 and size 2 (always a bad idea) it will
> pause IO in case of a failure until the recovery has finished. So the
> described behaviour is expected.
>
>
> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
>
> > Hi,
> >
> > Crush rule is "replicated" and min_size 2 actually. I am trying to
test
> > multiple volume configs in a single filesystem
> > using file layout.
> >
> > I have created metadata pool with rep 3 (min_size2 and replicated crush
> > rule) and data pool with rep 3 (min_size2 and replicated crush rule).
> and
> > also I have created multiple (replica 2, ec2-1 & ec4-2) pools and added
> to
> > the filesystem.
> >
> > Using file layout I have set different data pool to a different folders.
> so
> > I can test different configs in the same filesystem. all data pools
> > min_size set to handle single node failure.
> >
> > Single node failure is handled properly when only having metadata pool
> and
> > one data pool (rep3).
> >
> > After adding additional data pool to fs, single node failure scenario is
> > not working.
> >
> > regards
> > Amudhan P
> >
> > On Sun, May 17, 2020 at 1:29 AM Eugen Block <eblock(a)nde.ag> wrote:
> >
> >> What’s your pool configuration wrt min_size and crush rules?
> >>
> >>
> >> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
> >>
> >> > Hi,
> >> >
> >> > I am using ceph Nautilus cluster with below configuration.
> >> >
> >> > 3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr
are
> >> running
> >> > in shared mode.
> >> >
> >> > The client mounted through ceph kernel client.
> >> >
> >> > I was trying to emulate a node failure when a write and read were
> going
> >> on
> >> > (replica2) pool.
> >> >
> >> > I was expecting read and write continue after a small pause due to a
> Node
> >> > failure but it halts and never resumes until the failed node is up.
> >> >
> >> > I remember I tested the same scenario before in ceph mimic where it
> >> > continued IO after a small pause.
> >> >
> >> > regards
> >> > Amudhan P
> >> > _______________________________________________
> >> > ceph-users mailing list -- ceph-users(a)ceph.io
> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >>
> >>
> >>
> >>
>
>
>
>
24 May
24 May
7:25 a.m.
Sorry for the late reply.
I have pasted crush map in below url : https://pastebin.com/ASPpY2VB
and this my osd tree output and this issue are only when i use it with
filelayout.
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 327.48047 root default
-3 109.16016 host strgsrv01
0 hdd 5.45799 osd.0 up 1.00000 1.00000
2 hdd 5.45799 osd.2 up 1.00000 1.00000
3 hdd 5.45799 osd.3 up 1.00000 1.00000
4 hdd 5.45799 osd.4 up 1.00000 1.00000
5 hdd 5.45799 osd.5 up 1.00000 1.00000
6 hdd 5.45799 osd.6 up 1.00000 1.00000
7 hdd 5.45799 osd.7 up 1.00000 1.00000
19 hdd 5.45799 osd.19 up 1.00000 1.00000
20 hdd 5.45799 osd.20 up 1.00000 1.00000
21 hdd 5.45799 osd.21 up 1.00000 1.00000
22 hdd 5.45799 osd.22 up 1.00000 1.00000
23 hdd 5.45799 osd.23 up 1.00000 1.00000
-5 109.16016 host strgsrv02
1 hdd 5.45799 osd.1 up 1.00000 1.00000
8 hdd 5.45799 osd.8 up 1.00000 1.00000
9 hdd 5.45799 osd.9 up 1.00000 1.00000
10 hdd 5.45799 osd.10 up 1.00000 1.00000
11 hdd 5.45799 osd.11 up 1.00000 1.00000
12 hdd 5.45799 osd.12 up 1.00000 1.00000
24 hdd 5.45799 osd.24 up 1.00000 1.00000
25 hdd 5.45799 osd.25 up 1.00000 1.00000
26 hdd 5.45799 osd.26 up 1.00000 1.00000
27 hdd 5.45799 osd.27 up 1.00000 1.00000
28 hdd 5.45799 osd.28 up 1.00000 1.00000
29 hdd 5.45799 osd.29 up 1.00000 1.00000
-7 109.16016 host strgsrv03
13 hdd 5.45799 osd.13 up 1.00000 1.00000
14 hdd 5.45799 osd.14 up 1.00000 1.00000
15 hdd 5.45799 osd.15 up 1.00000 1.00000
16 hdd 5.45799 osd.16 up 1.00000 1.00000
17 hdd 5.45799 osd.17 up 1.00000 1.00000
18 hdd 5.45799 osd.18 up 1.00000 1.00000
30 hdd 5.45799 osd.30 up 1.00000 1.00000
31 hdd 5.45799 osd.31 up 1.00000 1.00000
32 hdd 5.45799 osd.32 up 1.00000 1.00000
33 hdd 5.45799 osd.33 up 1.00000 1.00000
34 hdd 5.45799 osd.34 up 1.00000 1.00000
35 hdd 5.45799 osd.35 up 1.00000 1.00000
On Tue, May 19, 2020 at 12:16 PM Eugen Block <eblock(a)nde.ag> wrote:
Was that a typo and you mean you changed min_size to
1? I/O paus with
min_size 1 and size 2 is unexpected, can you share more details like
your crushmap and your osd tree?
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
Behaviour is same even after setting min_size 2.
On Mon 18 May, 2020, 12:34 PM Eugen Block, <eblock(a)nde.ag> wrote:
> If your pool has a min_size 2 and size 2 (always a bad idea) it will
> pause IO in case of a failure until the recovery has finished. So the
> described behaviour is expected.
>
>
> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
>
> > Hi,
> >
> > Crush rule is "replicated" and min_size 2 actually. I am trying to
test
> > multiple volume configs in a single
filesystem
> > using file layout.
> >
> > I have created metadata pool with rep 3 (min_size2 and replicated
crush
> > rule) and data pool with rep 3
(min_size2 and replicated crush rule).
> and
> > also I have created multiple (replica 2, ec2-1 & ec4-2) pools and
added
> to
> > the filesystem.
> >
> > Using file layout I have set different data pool to a different
folders.
> so
> > I can test different configs in the same filesystem. all data pools
> > min_size set to handle single node failure.
> >
> > Single node failure is handled properly when only having metadata pool
> and
> > one data pool (rep3).
> >
> > After adding additional data pool to fs, single node failure scenario
is
> > not working.
> >
> > regards
> > Amudhan P
> >
> > On Sun, May 17, 2020 at 1:29 AM Eugen Block <eblock(a)nde.ag> wrote:
> >
> >> What’s your pool configuration wrt min_size and crush rules?
> >>
> >>
> >> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
> >>
> >> > Hi,
> >> >
> >> > I am using ceph Nautilus cluster with below configuration.
> >> >
> >> > 3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr
are
> >> running
> >> > in shared mode.
> >> >
> >> > The client mounted through ceph kernel client.
> >> >
> >> > I was trying to emulate a node failure when a write and read were
> going
> >> on
> >> > (replica2) pool.
> >> >
> >> > I was expecting read and write continue after a small pause due to
a
> Node
> >> > failure but it halts and never resumes until the failed node is up.
> >> >
> >> > I remember I tested the same scenario before in ceph mimic where it
> >> > continued IO after a small pause.
> >> >
> >> > regards
> >> > Amudhan P
> >> > _______________________________________________
> >> > ceph-users mailing list -- ceph-users(a)ceph.io
> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >>
> >>
> >>
> >>
>
>
>
>
25 May
25 May
2:48 p.m.
Hi,
you didn't really clear things up so I'll just summerarize what I
understood so far. Please also share 'ceph osd pool ls detail' and
'ceph fs status'.
One of the pools is configured with min_size 2 and size 2, this will
pause IO if one node goes down as it's very likely that this node
contains PGs from that pool. The IO will resume as soon as those PGs
are recovered on the remaining nodes. To serve IO despite the failure
you can set min_size to 1 for that pool.
If you take down the node with the active mds the failover can take
some time, of course.
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
Sorry for the late reply.
I have pasted crush map in below url : https://pastebin.com/ASPpY2VB
and this my osd tree output and this issue are only when i use it with
filelayout.
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 327.48047 root default
-3 109.16016 host strgsrv01
0 hdd 5.45799 osd.0 up 1.00000 1.00000
2 hdd 5.45799 osd.2 up 1.00000 1.00000
3 hdd 5.45799 osd.3 up 1.00000 1.00000
4 hdd 5.45799 osd.4 up 1.00000 1.00000
5 hdd 5.45799 osd.5 up 1.00000 1.00000
6 hdd 5.45799 osd.6 up 1.00000 1.00000
7 hdd 5.45799 osd.7 up 1.00000 1.00000
19 hdd 5.45799 osd.19 up 1.00000 1.00000
20 hdd 5.45799 osd.20 up 1.00000 1.00000
21 hdd 5.45799 osd.21 up 1.00000 1.00000
22 hdd 5.45799 osd.22 up 1.00000 1.00000
23 hdd 5.45799 osd.23 up 1.00000 1.00000
-5 109.16016 host strgsrv02
1 hdd 5.45799 osd.1 up 1.00000 1.00000
8 hdd 5.45799 osd.8 up 1.00000 1.00000
9 hdd 5.45799 osd.9 up 1.00000 1.00000
10 hdd 5.45799 osd.10 up 1.00000 1.00000
11 hdd 5.45799 osd.11 up 1.00000 1.00000
12 hdd 5.45799 osd.12 up 1.00000 1.00000
24 hdd 5.45799 osd.24 up 1.00000 1.00000
25 hdd 5.45799 osd.25 up 1.00000 1.00000
26 hdd 5.45799 osd.26 up 1.00000 1.00000
27 hdd 5.45799 osd.27 up 1.00000 1.00000
28 hdd 5.45799 osd.28 up 1.00000 1.00000
29 hdd 5.45799 osd.29 up 1.00000 1.00000
-7 109.16016 host strgsrv03
13 hdd 5.45799 osd.13 up 1.00000 1.00000
14 hdd 5.45799 osd.14 up 1.00000 1.00000
15 hdd 5.45799 osd.15 up 1.00000 1.00000
16 hdd 5.45799 osd.16 up 1.00000 1.00000
17 hdd 5.45799 osd.17 up 1.00000 1.00000
18 hdd 5.45799 osd.18 up 1.00000 1.00000
30 hdd 5.45799 osd.30 up 1.00000 1.00000
31 hdd 5.45799 osd.31 up 1.00000 1.00000
32 hdd 5.45799 osd.32 up 1.00000 1.00000
33 hdd 5.45799 osd.33 up 1.00000 1.00000
34 hdd 5.45799 osd.34 up 1.00000 1.00000
35 hdd 5.45799 osd.35 up 1.00000 1.00000
On Tue, May 19, 2020 at 12:16 PM Eugen Block <eblock(a)nde.ag> wrote:
> Was that a typo and you mean you changed min_size to 1? I/O paus with
> min_size 1 and size 2 is unexpected, can you share more details like
> your crushmap and your osd tree?
>
>
> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
>
> > Behaviour is same even after setting min_size 2.
> >
> > On Mon 18 May, 2020, 12:34 PM Eugen Block, <eblock(a)nde.ag> wrote:
> >
> >> If your pool has a min_size 2 and size 2 (always a bad idea) it will
> >> pause IO in case of a failure until the recovery has finished. So the
> >> described behaviour is expected.
> >>
> >>
> >> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
> >>
> >> > Hi,
> >> >
> >> > Crush rule is "replicated" and min_size 2 actually. I am
trying to
> test
> >> > multiple volume configs in a single filesystem
> >> > using file layout.
> >> >
> >> > I have created metadata pool with rep 3 (min_size2 and replicated
> crush
> >> > rule) and data pool with rep 3 (min_size2 and replicated crush rule).
> >> and
> >> > also I have created multiple (replica 2, ec2-1 & ec4-2) pools and
> added
> >> to
> >> > the filesystem.
> >> >
> >> > Using file layout I have set different data pool to a different
> folders.
> >> so
> >> > I can test different configs in the same filesystem. all data pools
> >> > min_size set to handle single node failure.
> >> >
> >> > Single node failure is handled properly when only having metadata pool
> >> and
> >> > one data pool (rep3).
> >> >
> >> > After adding additional data pool to fs, single node failure scenario
> is
> >> > not working.
> >> >
> >> > regards
> >> > Amudhan P
> >> >
> >> > On Sun, May 17, 2020 at 1:29 AM Eugen Block <eblock(a)nde.ag>
wrote:
> >> >
> >> >> What’s your pool configuration wrt min_size and crush rules?
> >> >>
> >> >>
> >> >> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I am using ceph Nautilus cluster with below configuration.
> >> >> >
> >> >> > 3 node's (Ubuntu 18.04) each has 12 OSD's, and mds,
mon and mgr are
> >> >> running
> >> >> > in shared mode.
> >> >> >
> >> >> > The client mounted through ceph kernel client.
> >> >> >
> >> >> > I was trying to emulate a node failure when a write and read
were
> >> going
> >> >> on
> >> >> > (replica2) pool.
> >> >> >
> >> >> > I was expecting read and write continue after a small pause
due to
> a
> >> Node
> >> >> > failure but it halts and never resumes until the failed node
is up.
> >> >> >
> >> >> > I remember I tested the same scenario before in ceph mimic
where it
> >> >> > continued IO after a small pause.
> >> >> >
> >> >> > regards
> >> >> > Amudhan P
> >> >> > _______________________________________________
> >> >> > ceph-users mailing list -- ceph-users(a)ceph.io
> >> >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
> >>
>
>
>
>
3:14 p.m.
Hello,
Sorry for the late reply.
I have pasted crush map in below url : https://pastebin.com/ASPpY2VB
and this my osd tree output and this issue are only when i use it with
filelayout.
could send the output of "ceph osd pool ls detail" please ?
Yoann
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT
PRI-AFF
-1 327.48047 root default
-3 109.16016 host strgsrv01
0 hdd 5.45799 osd.0 up 1.00000 1.00000
2 hdd 5.45799 osd.2 up 1.00000 1.00000
3 hdd 5.45799 osd.3 up 1.00000 1.00000
4 hdd 5.45799 osd.4 up 1.00000 1.00000
5 hdd 5.45799 osd.5 up 1.00000 1.00000
6 hdd 5.45799 osd.6 up 1.00000 1.00000
7 hdd 5.45799 osd.7 up 1.00000 1.00000
19 hdd 5.45799 osd.19 up 1.00000 1.00000
20 hdd 5.45799 osd.20 up 1.00000 1.00000
21 hdd 5.45799 osd.21 up 1.00000 1.00000
22 hdd 5.45799 osd.22 up 1.00000 1.00000
23 hdd 5.45799 osd.23 up 1.00000 1.00000
-5 109.16016 host strgsrv02
1 hdd 5.45799 osd.1 up 1.00000 1.00000
8 hdd 5.45799 osd.8 up 1.00000 1.00000
9 hdd 5.45799 osd.9 up 1.00000 1.00000
10 hdd 5.45799 osd.10 up 1.00000 1.00000
11 hdd 5.45799 osd.11 up 1.00000 1.00000
12 hdd 5.45799 osd.12 up 1.00000 1.00000
24 hdd 5.45799 osd.24 up 1.00000 1.00000
25 hdd 5.45799 osd.25 up 1.00000 1.00000
26 hdd 5.45799 osd.26 up 1.00000 1.00000
27 hdd 5.45799 osd.27 up 1.00000 1.00000
28 hdd 5.45799 osd.28 up 1.00000 1.00000
29 hdd 5.45799 osd.29 up 1.00000 1.00000
-7 109.16016 host strgsrv03
13 hdd 5.45799 osd.13 up 1.00000 1.00000
14 hdd 5.45799 osd.14 up 1.00000 1.00000
15 hdd 5.45799 osd.15 up 1.00000 1.00000
16 hdd 5.45799 osd.16 up 1.00000 1.00000
17 hdd 5.45799 osd.17 up 1.00000 1.00000
18 hdd 5.45799 osd.18 up 1.00000 1.00000
30 hdd 5.45799 osd.30 up 1.00000 1.00000
31 hdd 5.45799 osd.31 up 1.00000 1.00000
32 hdd 5.45799 osd.32 up 1.00000 1.00000
33 hdd 5.45799 osd.33 up 1.00000 1.00000
34 hdd 5.45799 osd.34 up 1.00000 1.00000
35 hdd 5.45799 osd.35 up 1.00000 1.00000
On Tue, May 19, 2020 at 12:16 PM Eugen Block <eblock(a)nde.ag> wrote:
--
Yoann Moulin
EPFL IC-IT
Was that a typo and you mean you changed min_size
to 1? I/O paus with
min_size 1 and size 2 is unexpected, can you share more details like
your crushmap and your osd tree?
Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
Behaviour is same even after setting min_size 2.
On Mon 18 May, 2020, 12:34 PM Eugen Block, <eblock(a)nde.ag> wrote:
> If your pool has a min_size 2 and size 2 (always a bad idea) it will
> pause IO in case of a failure until the recovery has finished. So the
> described behaviour is expected.
>
>
> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
>
>> Hi,
>>
>> Crush rule is "replicated" and min_size 2 actually. I am trying to
test
>> multiple volume configs in a single
filesystem
>> using file layout.
>>
>> I have created metadata pool with rep 3 (min_size2 and replicated
crush
>> rule) and data pool with rep 3
(min_size2 and replicated crush rule).
> and
>> also I have created multiple (replica 2, ec2-1 & ec4-2) pools and
added
> to
>> the filesystem.
>>
>> Using file layout I have set different data pool to a different
folders.
> so
>> I can test different configs in the same filesystem. all data pools
>> min_size set to handle single node failure.
>>
>> Single node failure is handled properly when only having metadata pool
> and
>> one data pool (rep3).
>>
>> After adding additional data pool to fs, single node failure scenario
is
>> not working.
>>
>> regards
>> Amudhan P
>>
>> On Sun, May 17, 2020 at 1:29 AM Eugen Block <eblock(a)nde.ag> wrote:
>>
>>> What’s your pool configuration wrt min_size and crush rules?
>>>
>>>
>>> Zitat von Amudhan P <amudhan83(a)gmail.com>om>:
>>>
>>>> Hi,
>>>>
>>>> I am using ceph Nautilus cluster with below configuration.
>>>>
>>>> 3 node's (Ubuntu 18.04) each has 12 OSD's, and mds, mon and mgr
are
>>> running
>>>> in shared mode.
>>>>
>>>> The client mounted through ceph kernel client.
>>>>
>>>> I was trying to emulate a node failure when a write and read were
> going
>>> on
>>>> (replica2) pool.
>>>>
>>>> I was expecting read and write continue after a small pause due to
a
> Node
>>>> failure but it halts and never resumes until the failed node is up.
>>>>
>>>> I remember I tested the same scenario before in ceph mimic where it
>>>> continued IO after a small pause.
>>>>
>>>> regards
>>>> Amudhan P
>>>> _______________________________________________
>>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
>>>
>>>
>>>
>
>
>
>
1451
days inactive
1460
days old
8 comments
3 participants
participants (3)
-
Amudhan P -
Eugen Block -
Yoann Moulin