4 Sep
2019
4 Sep
'19
7:35 p.m.
Hi everyone:
I asked about this on the #ceph IRC channel, but didn't get much traction (and, as an
aside: the advertised host for the channel's logs turns up unaccessible to me...).
I have a new Ceph cluster presenting an erasure-coded pool.
Current configuration is 8 nodes, each hosting a mon, and 1 osd per hdd (20 10TB HGST
spinning disks per node) for a total of 160 OSDs in the cluster.
The cluster configures fine with ceph-ansible (as nautilus or mimic), and ceph health is
always marked as good (other than the "warn" for not having associated an
application string with the current test pool).
rados bench maxes out the bandwidth of the network interface when I try it with 4MB
objects.
However, attempting a more "real-world" test of
rados -p ecpool --striper put obj 120MBfile
causes the transfer to fail with "operation not permitted (95)"
Inspection reveals that 3 stripe chunks get created - the first being the expected size,
and the second and third being only a few kb in size.
Object metadata from rados -p --striper ls obj is inconsistent with the sum of the on-disk
size of the chunks.
Can you advise how to diagnose what's breaking here?
Thanks
Sam Skipsey
University of Glasgow
4 Sep
4 Sep
9:52 p.m.
Additional info:
Trying this with a bog-standard replicated pool
(ceph osd pool create reppool 2048 2048)
succeeds perfectly.
So, I assume the issue is the write-amplification from the ec pool implementation?
My current settings are k=10, m=2
Reading in the list, there are additional suggestions about setting k to powers-of-two
(which aren't well documented in the official docs...).
5 Sep
5 Sep
1:11 a.m.
I think I can replicate your issue on a luminous cluster. It works fine with a 8+3 pool,
but 10+2 fails after creating the 3 chunks with the same error.
What does your erasure code profile look like? I don't think I actually tested
anything other than powers of two (8,16) for k before settling on 8.
Have you or can you check if 8+3 has the same behaviour? I may be missing something really
obvious here, but I haven't thought about this for a while.
Cheers,
Tom
-----Original Message-----
From: aoanla(a)gmail.com <aoanla(a)gmail.com>
Sent: 04 September 2019 11:36
To: ceph-users(a)ceph.io
Subject: [ceph-users] rados + radosstriper puts fail with "large" input objects
(mimic/nautilus, ec pool)
Hi everyone:
I asked about this on the #ceph IRC channel, but didn't get much traction (and, as an
aside: the advertised host for the channel's logs turns up unaccessible to me...).
I have a new Ceph cluster presenting an erasure-coded pool.
Current configuration is 8 nodes, each hosting a mon, and 1 osd per hdd (20 10TB HGST
spinning disks per node) for a total of 160 OSDs in the cluster.
The cluster configures fine with ceph-ansible (as nautilus or mimic), and ceph health is
always marked as good (other than the "warn" for not having associated an
application string with the current test pool).
rados bench maxes out the bandwidth of the network interface when I try it with 4MB
objects.
However, attempting a more "real-world" test of
rados -p ecpool --striper put obj 120MBfile
causes the transfer to fail with "operation not permitted (95)"
Inspection reveals that 3 stripe chunks get created - the first being the expected size,
and the second and third being only a few kb in size.
Object metadata from rados -p --striper ls obj is inconsistent with the sum of the on-disk
size of the chunks.
Can you advise how to diagnose what's breaking here?
Thanks
Sam Skipsey
University of Glasgow
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
3:36 a.m.
Hi Tom:
So, the failing ec code is also 10+2 here:
ceph osd erasure-code-profile set glasgow-ec \
k=10 \
m=2 \
crush-failure-domain=osd
Test EC codes which have succeeded, so far are:
k=4, m=2 failure-domain=osd
k=4, m=2 failure-domain=host
k=8, m=2 failure-domain=osd
EC codes which have failed
k=6, m=2 failure-domain=osd
k=10, m=2 failure-domain=osd
It seems likely that jerasure ec pools (at least) are not compatible with libradosstriper
if k is not a power of 2.
I suggest this be submitted as a bug?
7:58 a.m.
It seems likely that jerasure ec pools (at least) are
not compatible with libradosstriper if k is not a power of 2.
Yes, that seems like the summary of the issue as it stands, I'm intrigued to know what
libradosstriper is doing compared to librados that makes it only work on specific layouts,
as in my mind these two layers were effectively separate, but there must be something
going on here that I'm not aware of.
This definitely seems like a bug, or at least incredibly confusing undocumented
behaviour.
Cheers,
Tom
-----Original Message-----
From: aoanla(a)gmail.com <aoanla(a)gmail.com>
Sent: 04 September 2019 19:36
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: rados + radosstriper puts fail with "large" input
objects (mimic/nautilus, ec pool)
Hi Tom:
So, the failing ec code is also 10+2 here:
ceph osd erasure-code-profile set glasgow-ec \
k=10 \
m=2 \
crush-failure-domain=osd
Test EC codes which have succeeded, so far are:
k=4, m=2 failure-domain=osd
k=4, m=2 failure-domain=host
k=8, m=2 failure-domain=osd
EC codes which have failed
k=6, m=2 failure-domain=osd
k=10, m=2 failure-domain=osd
It seems likely that jerasure ec pools (at least) are not compatible with libradosstriper
if k is not a power of 2.
I suggest this be submitted as a bug?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
5:02 p.m.
Hi,
I'm not sure if this could really be a bug. We have configured several
ec pools with different profiles, many of them with k not a power of
2, e.g. k3 m7, k7 m11 on Luminous, k5 m1 in a Nautilus test cluster, I
haven't had any issues regarding these profile yet.
Can you share the ruleset the pool is created with? Are all of the PGs
actually created? You can check with
ceph pg ls-by-pool <POOL> | tr -s ' ' | cut -d " " --fields=1,17
Regards,
Eugen
Zitat von Thomas Byrne - UKRI STFC <tom.byrne(a)stfc.ac.uk>uk>:
It seems
likely that jerasure ec pools (at least) are not
compatible with libradosstriper if k is not a power of 2.
Yes, that seems like the summary of the issue as it stands, I'm
intrigued to know what libradosstriper is doing compared to librados
that makes it only work on specific layouts, as in my mind these two
layers were effectively separate, but there must be something going
on here that I'm not aware of.
This definitely seems like a bug, or at least incredibly confusing
undocumented behaviour.
Cheers,
Tom
-----Original Message-----
From: aoanla(a)gmail.com <aoanla(a)gmail.com>
Sent: 04 September 2019 19:36
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: rados + radosstriper puts fail with
"large" input objects (mimic/nautilus, ec pool)
Hi Tom:
So, the failing ec code is also 10+2 here:
ceph osd erasure-code-profile set glasgow-ec \
k=10 \
m=2 \
crush-failure-domain=osd
Test EC codes which have succeeded, so far are:
k=4, m=2 failure-domain=osd
k=4, m=2 failure-domain=host
k=8, m=2 failure-domain=osd
EC codes which have failed
k=6, m=2 failure-domain=osd
k=10, m=2 failure-domain=osd
It seems likely that jerasure ec pools (at least) are not compatible
with libradosstriper if k is not a power of 2.
I suggest this be submitted as a bug?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:05 p.m.
The placement groups are created, and the pool is completely functional for non-striper
use. I can 'rados put' test objects in just fine to a pool created with any k
value.
It's when the --striper option in 'rados put' is used to invoke the use of
libradosstriper, it fails on pools where k is not a power of two.
Example working and non-working profiles are as follows (I also tried w=16 for the 10+3).
[root@ceph-dev-mon2 ~]# ceph osd erasure-code-profile get je-8-3-rfd-osd-echo-nodes
crush-device-class=hdd
crush-failure-domain=osd
crush-root=echo-nodes
jerasure-per-chunk-alignment=false
k=8
m=3
plugin=jerasure
technique=reed_sol_van
w=8
[root@ceph-dev-mon2 ~]# ceph osd erasure-code-profile get je-10-3-rfd-osd-echo-nodes-2
crush-device-class=hdd
crush-failure-domain=osd
crush-root=echo-nodes
jerasure-per-chunk-alignment=false
k=10
m=3
plugin=jerasure
technique=reed_sol_van
w=8
-----Original Message-----
From: Eugen Block <eblock(a)nde.ag>
Sent: 05 September 2019 09:02
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: rados + radosstriper puts fail with "large" input
objects (mimic/nautilus, ec pool)
Hi,
I'm not sure if this could really be a bug. We have configured several ec pools with
different profiles, many of them with k not a power of 2, e.g. k3 m7, k7 m11 on Luminous,
k5 m1 in a Nautilus test cluster, I haven't had any issues regarding these profile
yet.
Can you share the ruleset the pool is created with? Are all of the PGs actually created?
You can check with
ceph pg ls-by-pool <POOL> | tr -s ' ' | cut -d " " --fields=1,17
Regards,
Eugen
Zitat von Thomas Byrne - UKRI STFC <tom.byrne(a)stfc.ac.uk>uk>:
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
It seems
likely that jerasure ec pools (at least) are not compatible
with libradosstriper if k is not a power of 2.
Yes, that seems like the summary of the issue as it stands, I'm
intrigued to know what libradosstriper is doing compared to librados
that makes it only work on specific layouts, as in my mind these two
layers were effectively separate, but there must be something going on
here that I'm not aware of.
This definitely seems like a bug, or at least incredibly confusing
undocumented behaviour.
Cheers,
Tom
-----Original Message-----
From: aoanla(a)gmail.com <aoanla(a)gmail.com>
Sent: 04 September 2019 19:36
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: rados + radosstriper puts fail with "large"
input objects (mimic/nautilus, ec pool)
Hi Tom:
So, the failing ec code is also 10+2 here:
ceph osd erasure-code-profile set glasgow-ec \
k=10 \
m=2 \
crush-failure-domain=osd
Test EC codes which have succeeded, so far are:
k=4, m=2 failure-domain=osd
k=4, m=2 failure-domain=host
k=8, m=2 failure-domain=osd
EC codes which have failed
k=6, m=2 failure-domain=osd
k=10, m=2 failure-domain=osd
It seems likely that jerasure ec pools (at least) are not compatible
with libradosstriper if k is not a power of 2.
I suggest this be submitted as a bug?
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
1697
days inactive
1698
days old
6 comments
3 participants
participants (3)
-
aoanla＠gmail.com -
Eugen Block -
Thomas Byrne - UKRI STFC