[ceph-users] Re: Unsubscribe this mail list

Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics: 1. Re: Removing pool in nautilus is incredibly slow (Francois Legrand) 2. Re: Bench on specific OSD (Marc Roos) 3. Re: Removing pool in nautilus is incredibly slow (Wout van Heeswijk) 4. Lifecycle message on logs (Marcelo Miziara) 5. Re: Feedback of the used configuration (Simon Sutter) 6. Re: Removing pool in nautilus is incredibly slow (Francois Legrand) 7. Re: Removing pool in nautilus is incredibly slow (Eugen Block) ---------------------------------------------------------------------- Date: Thu, 25 Jun 2020 07:57:48 -0000 From: "Francois Legrand" &lt;fleg(a)lpnhe.in2p3.fr&gt; Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow To: ceph-users(a)ceph.io Message-ID: <159307186843.20.7354239610800797635@mailman-web> Content-Type: text/plain; charset="utf-8" Does someone have an idea ? F. ------------------------------ Date: Thu, 25 Jun 2020 11:36:24 +0200 From: "Marc Roos" &lt;M.Roos(a)f1-outsourcing.eu&gt; Subject: [ceph-users] Re: Bench on specific OSD To: ceph-users &lt;ceph-users(a)ceph.io&gt;io>, seenafallah &lt;seenafallah(a)gmail.com&gt; Message-ID: <"H0000071001729d1.1593077784.sx.f1-outsourcing.eu*"@MHS> Content-Type: text/plain; charset="US-ASCII" What is wrong with just doing multiple tests and group in your charts osd's by host? -----Original Message----- To: ceph-users Subject: [ceph-users] Bench on specific OSD Hi all. Is there anyway to completely health check one OSD host or instance? For example rados bech just on that OSD or do some checks for disk and front and back netowrk? Thanks. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io ------------------------------ Date: Thu, 25 Jun 2020 14:26:59 +0200 From: Wout van Heeswijk &lt;wout(a)42on.com&gt; Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow To: fleg(a)lpnhe.in2p3.fr Cc: ceph-users(a)ceph.io Message-ID: &lt;38a5d557-7797-a68a-1b67-4c8f0e1ecf4c(a)42on.com&gt; Content-Type: text/plain; charset=utf-8; format=flowed Hi Francois, Have you already looked at the option "osd_delete_sleep"? It will not speed up the process but I will give you some control over your cluster performance. Something like: ceph tell osd.\* injectargs '--osd_delete_sleep1' kind regards, Wout 42on On 25-06-2020 09:57, Francois Legrand wrote: ------------------------------ Date: Thu, 25 Jun 2020 09:53:09 -0300 From: Marcelo Miziara &lt;raxidex(a)gmail.com&gt; Subject: [ceph-users] Lifecycle message on logs To: ceph-users(a)ceph.io Message-ID: &lt;CAGxsqB296eczKux19OKObkimXg3PZ_UDEWVTTqEnrAf=Agm0+A(a)mail.gmail.com&gt; Content-Type: text/plain; charset="UTF-8" Hello...it's the first time I need to use the lifecycle, and I created a bucket and set it to expire in one day with s3cmd: s3cmd expire --expiry-days=1 s3://bucket The rgw_lifecycle_work_time is set to the default values(00:00-06:00). But I noticed in the rgw logs a lot of messages like: 2020-06-16 00:00:00.311369 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.8 2020-06-16 00:00:00.311623 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.311862 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.4 2020-06-16 00:00:00.319424 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.10 2020-06-16 00:00:00.319647 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.18 2020-06-16 00:00:00.320682 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.327770 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.6 2020-06-16 00:00:00.328941 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.17 2020-06-16 00:00:00.332463 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.20 2020-06-16 00:00:00.336788 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.1 2020-06-16 00:00:00.336924 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.24 2020-06-16 00:00:00.340915 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.2 The object was deleted, but these messages keep appearing. Is it safe to ignore them? For the records, i'm using redhat luminous 12.2.12 Thanks, Marcelo. ------------------------------ Date: Thu, 25 Jun 2020 13:19:38 +0000 From: Simon Sutter &lt;ssutter(a)hosttech.ch&gt; Subject: [ceph-users] Re: Feedback of the used configuration To: Paul Emmerich &lt;paul.emmerich(a)croit.io&gt; Cc: &quot;ceph-users(a)ceph.io&quot; &lt;ceph-users(a)ceph.io&gt; Message-ID: &lt;f6e11ee189a74c35815c495d64cbe4cb(a)hosttech.ch&gt; Content-Type: text/plain; charset="utf-8" Hello Paul, Thanks for the Answer. I took a look at the subvolumes, but they are a bit odd in my opinion. If I create one with a subvolume-group, the folder structure will look like this: /cephfs/volumes/group-name/subvolume-name/random-uuid/ And I have to issue two commands, first set the group and then set the volume name, but why so complicated? Wouldn't it be easier to just make subvolumes anywhere inside the cephfs? I can see the intended use for groups, but if I want to publish a pool in some different directory that's not possible (except for setfattr). Without first creating subvolume-groups, the orchestrator creates subvolumes in the /cephfs/volumes/_nogroup/subvolume-name/randmon-uuid/ folder. And the more important question is, why is there a new folder with a random uuid inside the subvolume? I try to understand the points the devs had, when they developed this, but for me, this is something I have to explain to some devs in our team and at the moment I can't. It is indeed easier to deploy but comes with much less flexibility. Maybe something to write in the tracker about? Thanks in advance, Simon Von: Paul Emmerich [mailto:paul.emmerich@croit.io] Gesendet: Mittwoch, 24. Juni 2020 17:35 An: Simon Sutter &lt;ssutter(a)hosttech.ch&gt; Cc: ceph-users(a)ceph.io Betreff: Re: [ceph-users] Feedback of the used configuration Have a look at cephfs subvolumes: https://docs.ceph.com/docs/master/cephfs/fs-volumes/#fs-subvolumes They are internally just directories with quota/pool placement layout/namespace with some mgr magic to make it easier than doing that all by hand Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90 On Wed, Jun 24, 2020 at 4:38 PM Simon Sutter <ssutter@hosttech.ch<mailto:ssutter@hosttech.ch>> wrote: Hello, After two months of the "ceph try and error game", I finally managed to get an Octopuss cluster up and running. The unconventional thing about it is, it's just for hot backups, no virtual machines on there. All the nodes are without any caching ssd's, just plain hdd's. At the moment there are eight of them with a total of 50TB. We are planning to go up to 25 and bigger disks so we end on 300TB-400TB I decided to go with cephfs, because I don't have any experience in things like S3 and I need to read the same file system from more than one client. I made one cephfs with a replicated pool. On there I added erasure-coded pools to save some Storage. To add those pools, I did it with the setfattr command like this: setfattr -n ceph.dir.layout.pool -v ec_data_server1 /cephfs/nfs/server1 Some of our servers cannot use cephfs (old kernels, special OS's) so I have to use nfs. This is set up with the included ganesha-nfs. Exported is the /cephfs/nfs folder and clients can mount folders below this. There are two final questions: - Was it right to go with the way of "mounting" pools with setfattr, or should I have used multiple cephfs? First I was thinking about using multiple cephfs but there are warnings everywhere. The deeper I got in, the more it seems I would have been fine with multiple cephfs. - Is there a way I don't know, but it would be easier? I still don't know much about Rest, S3, RBD etc... so there may be a better way Other remarks are desired. Thanks in advance, Simon _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io> ------------------------------ Date: Thu, 25 Jun 2020 16:22:12 +0200 From: Francois Legrand &lt;fleg(a)lpnhe.in2p3.fr&gt; Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow To: Wout van Heeswijk &lt;wout(a)42on.com&gt; Cc: ceph-users(a)ceph.io Message-ID: &lt;db34e791-1172-8b94-09e7-4ab5790d3162(a)lpnhe.in2p3.fr&gt; Content-Type: text/plain; charset=utf-8; format=flowed Thanks for the hint. I tryed but it doesn't seems to change anything... Moreover, as the osds seems quite loaded I had regularly some osd marked down which triggered some new peering and thus more load !!! I set the osd no down flag, but I still have some osd reported (wrongly) as down (and back up in the minute) which generate peering and remapping. I don't really understand the action of no down parameter ! Is there a way to tell ceph not to peer immediately after an osd is reported down (let say wait for 60s) ? I am thinking about restarting all osd (or maybe the whole cluster) to get osd_op_queue_cut_off changed to high and osd_op_thread_timeout to something higher than 15 (but I don't think it will really improve the situation). F. Le 25/06/2020 à 14:26, Wout van Heeswijk a écrit : ------------------------------ Date: Thu, 25 Jun 2020 14:42:57 +0000 From: Eugen Block &lt;eblock(a)nde.ag&gt; Subject: [ceph-users] Re: Removing pool in nautilus is incredibly slow To: ceph-users(a)ceph.io Message-ID: &lt;20200625144257.Horde.I8JPcddeor47WdOehQKQPNY(a)webmail.nde.ag&gt; Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes I'm not sure if your OSDs have their rocksDB on faster devices, if not it sounds a lot like rocksdb fragmentation [1] leading to a very high load on the OSDs and occasionally crashing OSDs. If you don't plan to delete so much data at once on a regular basis you could sit this one out, but one solution is to re-create the OSDs with rocksDB/WAL on faster devices. [1] https://www.mail-archive.com/ceph-users@ceph.io/msg03160.html Zitat von Francois Legrand &lt;fleg(a)lpnhe.in2p3.fr&gt;fr>: ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 89, Issue 112 *******************************************