- ceph-users - lists.ceph.io

librados behavior when some OSDs are unreachables
by David DELON 28 Jan '20

28 Jan '20

Hi, i had a problem with one application (seafile) which uses CEPH backend with librados. The corresponding pools are defined with size=3 and each object copy is on a different host. The cluster health is OK: all the monitors see all the hosts. Now, a network problem just happens between my RADOS client and a single host. Then, when my application/client tries to access an object which is situed on the unreachable host (primary for the corresponding PG), it does not failover to another copy/host (and my application crashes later because after a while, with many requests, too many files are opened on Linux). Is it the normal behavior? My storage is resilient (great!) but not its access... If on the host, i stop the OSDs or change the affinity to zero, it solves, so it seems like the librados just check and trust the osdmap And doing a tcpdump show the client tries to access the same OSD without timeout. It can be easily reproduced with defining a netfilter rule on a host to drop packets coming from the client. Note: i am still on Luminous (both on lient and cluster sides). Thanks for reading. D.

2 1

Question about erasure code
by Zorg 28 Jan '20

28 Jan '20

Hi, we are planning to use EC I have 3 questions about it 1 / what is the advantage of having more machines than (k + m)? We are planning to have 11 nodes and use k=8 and m=3. does it improved performance to have more node than K+M? of how many ? what ratio? 2 / what behavior should we expect if we lose 1,2 or 3 nodes: perfs / recovery. how many nodes could we loose before the cluster state change to read only 3 / what is the maximum occupancy rate on the EC? Thans for your help Zorg

2 1

Renaming LVM Groups of OSDs
by Stolte, Felix 28 Jan '20

28 Jan '20

Hi all, I would like to rename the logical volumes / volume groups used by my OSDs. Do I need to change anything else than the block and block.db links under /var/lib/ceph/osd/<osdid>? IT-Services Telefon 02461 61-9243 E-Mail: f.stolte(a)fz-juelich.de ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------

3 2

Re: EC pool creation results in incorrect M value?
by Geoffrey Rhodes 28 Jan '20

28 Jan '20

Hi Eric, With regards to "From the output of “ceph osd pool ls detail” you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. Am I missing something?" For your EC profile you need to set the pool min_size=3 to still read/write to the pool with two host failures. RUN: sudo ceph osd pool set ec32pool min_size 3 Kind regards Geoffrey Rhodes On Mon, 27 Jan 2020 at 22:11, <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: EC pool creation results in incorrect M value? (Paul Emmerich) > 2. Re: EC pool creation results in incorrect M value? (Smith, Eric) > 3. Re: EC pool creation results in incorrect M value? (Smith, Eric) > 4. data loss on full file system? (Håkan T Johansson) > > > ---------------------------------------------------------------------- > > Date: Mon, 27 Jan 2020 17:14:55 +0100 > From: Paul Emmerich <paul.emmerich(a)croit.io> > Subject: [ceph-users] Re: EC pool creation results in incorrect M > value? > To: "Smith, Eric" <Eric.Smith(a)ccur.com> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: > < > CAD9yTbFb28FX_XaNXRUUxQ1UqAYa_ouhO_fkE+Vbts1VXGkuDw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > min_size in the crush rule and min_size in the pool are completely > different things that happen to share the same name. > > Ignore min_size in the crush rule, it has virtually no meaning in > almost all cases (like this one). > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to > create an EC3+2 pool with the following commands: > > > > Create the EC profile: > > > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > > > Verify profile creation: > > > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > > > crush-device-class= > > > > crush-failure-domain=host > > > > crush-root=sgshared > > > > jerasure-per-chunk-alignment=false > > > > k=3 > > > > m=2 > > > > plugin=jerasure > > > > technique=reed_sol_van > > > > w=8 > > > > Create a pool using this profile: > > > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > > > List pool detail: > > > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > stripe_width 12288 application ES > > > > Here’s the crush rule that’s created: > > { > > > > "rule_id": 11, > > > > "rule_name": "es32", > > > > "ruleset": 11, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 5, > > > > "steps": [ > > > > { > > > > "op": "set_chooseleaf_tries", > > > > "num": 5 > > > > }, > > > > { > > > > "op": "set_choose_tries", > > > > "num": 100 > > > > }, > > > > { > > > > "op": "take", > > > > "item": -2, > > > > "item_name": "sgshared" > > > > }, > > > > { > > > > "op": "chooseleaf_indep", > > > > "num": 0, > > > > "type": "host" > > > > }, > > > > { > > > > "op": "emit" > > > > } > > > > ] > > > > }, > > > > > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the > crush rule says min_size=3 however the pool does NOT survive 2 hosts > failing. > > > > > > > > Am I missing something? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 27 Jan 2020 16:22:06 +0000 > From: "Smith, Eric" <Eric.Smith(a)ccur.com> > Subject: [ceph-users] Re: EC pool creation results in incorrect M > value? > To: Paul Emmerich <paul.emmerich(a)croit.io> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <BN8PR14MB28206D6D29507FD9499DEE91EA0B0(a)BN8PR14MB2820.nam > prd14.prod.outlook.com> > Content-Type: text/plain; charset="utf-8" > > Thanks for the info regarding min_size in the crush rule - does this seem > like a bug to you then? Is anyone else able to reproduce this? > > -----Original Message----- > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Monday, January 27, 2020 11:15 AM > To: Smith, Eric <Eric.Smith(a)ccur.com> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] EC pool creation results in incorrect M value? > > min_size in the crush rule and min_size in the pool are completely > different things that happen to share the same name. > > Ignore min_size in the crush rule, it has virtually no meaning in almost > all cases (like this one). > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to > create an EC3+2 pool with the following commands: > > > > Create the EC profile: > > > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > > > Verify profile creation: > > > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > > > crush-device-class= > > > > crush-failure-domain=host > > > > crush-root=sgshared > > > > jerasure-per-chunk-alignment=false > > > > k=3 > > > > m=2 > > > > plugin=jerasure > > > > technique=reed_sol_van > > > > w=8 > > > > Create a pool using this profile: > > > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > > > List pool detail: > > > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > > stripe_width 12288 application ES > > > > Here’s the crush rule that’s created: > > { > > > > "rule_id": 11, > > > > "rule_name": "es32", > > > > "ruleset": 11, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 5, > > > > "steps": [ > > > > { > > > > "op": "set_chooseleaf_tries", > > > > "num": 5 > > > > }, > > > > { > > > > "op": "set_choose_tries", > > > > "num": 100 > > > > }, > > > > { > > > > "op": "take", > > > > "item": -2, > > > > "item_name": "sgshared" > > > > }, > > > > { > > > > "op": "chooseleaf_indep", > > > > "num": 0, > > > > "type": "host" > > > > }, > > > > { > > > > "op": "emit" > > > > } > > > > ] > > > > }, > > > > > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the > crush rule says min_size=3 however the pool does NOT survive 2 hosts > failing. > > > > > > > > Am I missing something? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 27 Jan 2020 16:45:52 +0000 > From: "Smith, Eric" <Eric.Smith(a)ccur.com> > Subject: [ceph-users] Re: EC pool creation results in incorrect M > value? > To: "Smith, Eric" <Eric.Smith(a)ccur.com>, Paul Emmerich > <paul.emmerich(a)croit.io> > Cc: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <BN8PR14MB2820B1DF71DE395B7B5AA1ECEA0B0(a)BN8PR14MB2820.nam > prd14.prod.outlook.com> > Content-Type: text/plain; charset="utf-8" > > OK I see this: https://github.com/ceph/ceph/pull/8008 > > Perhaps it's just to be safe... > > -----Original Message----- > From: Smith, Eric <Eric.Smith(a)ccur.com> > Sent: Monday, January 27, 2020 11:22 AM > To: Paul Emmerich <paul.emmerich(a)croit.io> > Cc: ceph-users(a)ceph.io > Subject: [ceph-users] Re: EC pool creation results in incorrect M value? > > Thanks for the info regarding min_size in the crush rule - does this seem > like a bug to you then? Is anyone else able to reproduce this? > > -----Original Message----- > From: Paul Emmerich <paul.emmerich(a)croit.io> > Sent: Monday, January 27, 2020 11:15 AM > To: Smith, Eric <Eric.Smith(a)ccur.com> > Cc: ceph-users(a)ceph.io > Subject: Re: [ceph-users] EC pool creation results in incorrect M value? > > min_size in the crush rule and min_size in the pool are completely > different things that happen to share the same name. > > Ignore min_size in the crush rule, it has virtually no meaning in almost > all cases (like this one). > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to > create an EC3+2 pool with the following commands: > > > > Create the EC profile: > > > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > > > Verify profile creation: > > > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > > > crush-device-class= > > > > crush-failure-domain=host > > > > crush-root=sgshared > > > > jerasure-per-chunk-alignment=false > > > > k=3 > > > > m=2 > > > > plugin=jerasure > > > > technique=reed_sol_van > > > > w=8 > > > > Create a pool using this profile: > > > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > > > List pool detail: > > > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > > stripe_width 12288 application ES > > > > Here’s the crush rule that’s created: > > { > > > > "rule_id": 11, > > > > "rule_name": "es32", > > > > "ruleset": 11, > > > > "type": 3, > > > > "min_size": 3, > > > > "max_size": 5, > > > > "steps": [ > > > > { > > > > "op": "set_chooseleaf_tries", > > > > "num": 5 > > > > }, > > > > { > > > > "op": "set_choose_tries", > > > > "num": 100 > > > > }, > > > > { > > > > "op": "take", > > > > "item": -2, > > > > "item_name": "sgshared" > > > > }, > > > > { > > > > "op": "chooseleaf_indep", > > > > "num": 0, > > > > "type": "host" > > > > }, > > > > { > > > > "op": "emit" > > > > } > > > > ] > > > > }, > > > > > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the > crush rule says min_size=3 however the pool does NOT survive 2 hosts > failing. > > > > > > > > Am I missing something? > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Mon, 27 Jan 2020 21:10:10 +0100 > From: Håkan T Johansson <f96hajo(a)chalmers.se> > Subject: [ceph-users] data loss on full file system? > To: <ceph-users(a)ceph.io> > Message-ID: > <alpine.DEB.2.20.2001272106210.7767(a)planck-o.fy.chalmers.se> > Content-Type: text/plain; format=flowed; charset="UTF-8" > > > Hi, > > for test purposes, I have set up two 100 GB OSDs, one > taking a data pool and the other metadata pool for cephfs. > > Am running 14.2.6-1-gffd69200ad-1 with packages from > https://mirror.croit.io/debian-nautilus > > Am then running a program that creates a lot of 1 MiB files by calling > fopen() > fwrite() > fclose() > for each of them. Error codes are checked. > > This works successfully for ~100 GB of data, and then strangely also > succeeds > for many more 100 GB of data... ?? > > All written files have size 1 MiB with 'ls', and thus should contain the > data > written. However, on inspection, the files written after the first ~100 > GiB, > are full of just 0s. (hexdump -C) > > > To further test this, I use the standard tool 'cp' to copy a few > random-content > files into the full cephfs filessystem. cp reports no complaints, and > after > the copy operations, content is seen with hexdump -C. However, after > forcing > the data out of cache on the client by reading other earlier created > files, > hexdump -C show all-0 content for the files copied with 'cp'. Data that > was > there is suddenly gone...? > > > I am new to ceph. Is there an option I have missed to avoid this > behaviour? > (I could not find one in > https://docs.ceph.com/docs/master/man/8/mount.ceph/ ) > > Is this behaviour related to > https://docs.ceph.com/docs/mimic/cephfs/full/ > ? > > (That page states 'sometime after a write call has already returned 0'. > But if > write returns 0, then no data has been written, so the user program would > not > assume any kind of success.) > > Best regards, > > Håkan > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 84, Issue 44 > ****************************************** >

1 0

Re: EC pool creation results in incorrect M value?
by Smith, Eric 27 Jan '20

27 Jan '20

OK I see this: https://github.com/ceph/ceph/pull/8008 Perhaps it's just to be safe... -----Original Message----- From: Smith, Eric <Eric.Smith(a)ccur.com> Sent: Monday, January 27, 2020 11:22 AM To: Paul Emmerich <paul.emmerich(a)croit.io> Cc: ceph-users(a)ceph.io Subject: [ceph-users] Re: EC pool creation results in incorrect M value? Thanks for the info regarding min_size in the crush rule - does this seem like a bug to you then? Is anyone else able to reproduce this? -----Original Message----- From: Paul Emmerich <paul.emmerich(a)croit.io> Sent: Monday, January 27, 2020 11:15 AM To: Smith, Eric <Eric.Smith(a)ccur.com> Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] EC pool creation results in incorrect M value? min_size in the crush rule and min_size in the pool are completely different things that happen to share the same name. Ignore min_size in the crush rule, it has virtually no meaning in almost all cases (like this one). Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to create an EC3+2 pool with the following commands: > > Create the EC profile: > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 > technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > Verify profile creation: > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > crush-device-class= > > crush-failure-domain=host > > crush-root=sgshared > > jerasure-per-chunk-alignment=false > > k=3 > > m=2 > > plugin=jerasure > > technique=reed_sol_van > > w=8 > > Create a pool using this profile: > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > List pool detail: > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash > rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool > stripe_width 12288 application ES > > Here’s the crush rule that’s created: > { > > "rule_id": 11, > > "rule_name": "es32", > > "ruleset": 11, > > "type": 3, > > "min_size": 3, > > "max_size": 5, > > "steps": [ > > { > > "op": "set_chooseleaf_tries", > > "num": 5 > > }, > > { > > "op": "set_choose_tries", > > "num": 100 > > }, > > { > > "op": "take", > > "item": -2, > > "item_name": "sgshared" > > }, > > { > > "op": "chooseleaf_indep", > > "num": 0, > > "type": "host" > > }, > > { > > "op": "emit" > > } > > ] > > }, > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. > > > > Am I missing something? > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: EC pool creation results in incorrect M value?
by Paul Emmerich 27 Jan '20

27 Jan '20

min_size in the crush rule and min_size in the pool are completely different things that happen to share the same name. Ignore min_size in the crush rule, it has virtually no meaning in almost all cases (like this one). Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Jan 27, 2020 at 3:41 PM Smith, Eric <Eric.Smith(a)ccur.com> wrote: > > I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I’m attempting to create an EC3+2 pool with the following commands: > > Create the EC profile: > > ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 technique=reed_sol_van crush-failure-domain=host crush-root=sgshared > > Verify profile creation: > > [root@mon-1 ~]# ceph osd erasure-code-profile get es32 > > crush-device-class= > > crush-failure-domain=host > > crush-root=sgshared > > jerasure-per-chunk-alignment=false > > k=3 > > m=2 > > plugin=jerasure > > technique=reed_sol_van > > w=8 > > Create a pool using this profile: > > ceph osd pool create ec32pool 1024 1024 erasure es32 > > List pool detail: > > pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool stripe_width 12288 application ES > > Here’s the crush rule that’s created: > { > > "rule_id": 11, > > "rule_name": "es32", > > "ruleset": 11, > > "type": 3, > > "min_size": 3, > > "max_size": 5, > > "steps": [ > > { > > "op": "set_chooseleaf_tries", > > "num": 5 > > }, > > { > > "op": "set_choose_tries", > > "num": 100 > > }, > > { > > "op": "take", > > "item": -2, > > "item_name": "sgshared" > > }, > > { > > "op": "chooseleaf_indep", > > "num": 0, > > "type": "host" > > }, > > { > > "op": "emit" > > } > > ] > > }, > > > > From the output of “ceph osd pool ls detail” you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. > > > > Am I missing something? > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

EC pool creation results in incorrect M value?
by Smith, Eric 27 Jan '20

27 Jan '20

I have a Ceph Luminous (12.2.12) cluster with 6 nodes. I'm attempting to create an EC3+2 pool with the following commands: 1. Create the EC profile: * ceph osd erasure-code-profile set es32 k=3 m=2 plugin=jerasure w=8 technique=reed_sol_van crush-failure-domain=host crush-root=sgshared 2. Verify profile creation: [root@mon-1 ~]# ceph osd erasure-code-profile get es32 crush-device-class= crush-failure-domain=host crush-root=sgshared jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 1. Create a pool using this profile: * ceph osd pool create ec32pool 1024 1024 erasure es32 2. List pool detail: * pool 31 'es32' erasure size 5 min_size 4 crush_rule 11 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 1568 flags hashpspool stripe_width 12288 application ES 3. Here's the crush rule that's created: { "rule_id": 11, "rule_name": "es32", "ruleset": 11, "type": 3, "min_size": 3, "max_size": 5, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "sgshared" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }, From the output of "ceph osd pool ls detail" you can see min_size=4, the crush rule says min_size=3 however the pool does NOT survive 2 hosts failing. Am I missing something?

1 0

Re: Ceph MDS randomly hangs with no useful error message
by janek.bevendorff＠uni-weimar.de 26 Jan '20

26 Jan '20

1 0

How to accelerate deep scrub effectively?
by 徐蕴 25 Jan '20

25 Jan '20

Hi, I’ve just upgrade my cluster from Jewel to Nautilus (still running filestore). Since I'v stopped deepscrub for several days for this upgrade, now I got warning of "3 pgs not deep-scrubbed in time”. I tried to increase osd_max_scrubs to 3, osd_scrub_load_threshold to 5.0 and osd_deep_scrub_stride to 1048576 in order to accelerate the deepscrub processes. But deep scrubs seem still very slow. Is there any other way to improve the deep scrub performance? The cluster has 8 nodes, 52 HDD OSDs, 168 TiB / 313 TiB avail data, cluster traffic is low at present. BR, Xu Yun

1 0

Re: Unable to track different ceph client version connections
by Konstantin Shalygin 25 Jan '20

25 Jan '20

> We upgraded our Ceph cluster from Hammer to Luminous and it is running > fine. Post upgrade we live migrated all our Openstack instances (not 100% > sure). Currently we see 1658 clients still on Hammer version. To track the > clients we increased the debugging of debug_mon=10/10, debug_ms=1/5, > debug_monc=5/20 on all three monitors and looking at all three monitor logs > at /var/log/ceph/mon.<id>.log and grepping for hammer and 0x81dff8eeacfffb but > not seeing anything in logs even after hours of waiting. > > Earlier in our other clusters it used to show in logs from which Openstack > compute node it is originating from. Am I missing something or do I need > to add more logging or need to check a different log on all three ceph > monitor nodes? Look your clients at mon sessions: `ceph daemon /var/run/ceph/ceph-mon.ceph-mon0.asok sessions | grep hammer | awk '{print $2}'` k

2 1