- ceph-users - lists.ceph.io

PG lock contention? CephFS metadata pool rebalance
by Stefan Kooman 21 Jan '20

21 Jan '20

Hi, Like I said in an earlier mail to this list, we re-balanced ~ 60% of the CephFS metadata pool to NVMe backed devices. Roughly 422 M objects (1.2 Billion replicated). We have 512 PGs allocated to them. While rebalancing we suffered from quite a few SLOW_OPS. Memory, CPU and device IOPS capacity were not a limiting factor as far as we can see (plenty of them available ... nowhere near max capacity). We saw quite a few slow ops with the following events: "time": "2019-12-19 09:41:02.712010", "event": "reached_pg" }, { "time": "2019-12-19 09:41:02.712014", "event": "waiting for rw locks" }, { "time": "2019-12-19 09:41:02.881939", "event": "reached_pg" ... and this repeated 100's of times taking ~ 30 seconds to complete Does this indicate PG lock contention? If so ... would we need to provide more PGs to the metadata pool to avoid this? The metadata pool is only ~ 166 MiB big ... but with loads of OMAPs ... Most advice on PG planning is concerned with the _amount_ of data ... but the metadata pool (and this might also be true for RGW index pools) seem to be a special case. Thanks for your insights, Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

1 1

deep-scrub / backfilling: large amount of SLOW_OPS after upgrade to 13.2.8
by Stefan Kooman 21 Jan '20

21 Jan '20

Hi, After the upgrade to 13.2.8 deep-scrub has a big impact on client IO: loads of SLOW_OPS and high latency. We hardly ever had SLOW_OPS, but since the upgrade the impact is so big that we even have OSDs marking each other out (OSD op thread timeout) multiple times during the scrub window. Plenty of CPU / RAM / IOPS left, hardly any load on these OSD servers. Has there anything changed in this release that can explain this behaviour? Besides this the impact of rebalance is very severe as well. With only the balancer remapping a couple of PGs at a time there are loads of (MDS_)SLOW_OPS. This morning the cephfs metadata pool got rebalanced ... and that triggered a lot of SLOW_OPS. One particular OSD was pegged at 1000% CPU for more than half an hour (not doing that much IO): that's 10 cores going full throttle! After a restart this issue was gone. Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 3

subscribe
by lists 21 Jan '20

21 Jan '20

1 0

Upgrade from Jewel to Luminous resulted 82% misplacement
by 徐蕴 21 Jan '20

21 Jan '20

Hi, We upgraded our cluster from Jewel to Luminous, and it turned out that more than 80% object misplaced. Since our cluster has 130T data, backfilling seems take forever. We didn’t modify any crushmap. Any thoughts about this issue? br, Xu Yun

2 2

lists and gmail
by Sasha Litvak 20 Jan '20

20 Jan '20

It seems that people now split between new and old list servers. Regardless of either one of them, I am missing a number of messages that appear on archive pages but never seem to make to my inbox. And no they are not in my junk folder. I wonder if some of my questions are not getting a response because people don't receive them. Any other reason, like people choosing not to answer, is pretty acceptable. Does anyone else have difficulty communicating with user list using gmail account?

1 0

Re: cephfs kernel mount option uid?
by Kevin Thorpe 20 Jan '20

20 Jan '20

Hi, Pretty certain not. I hit that exact issue. The workaround I was suggested to use was an init container running as root to change the ownership. That works ok but is very hacky. -- *Kevin Thorpe* VP of Enterprise Platform *W* *|* www.predictx.com *P * *|* +44 (0)20 3005 6750 <+44%2020%203005%206750> | +44 (0)808 204 0344 <+44%20808%20204%200344> *A* *|* 7th Floor, 52 Grosvenor Gardens, London SW1W 0AU <https://maps.google.com/?q=7th+Floor,+52+Grosvenor+Gardens,+London+SW1W+0AU…> <http://twitter.com/predictxai> <https://www.linkedin.com/company/1339603/> _________________________________________________________ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. On Mon, 20 Jan 2020 at 16:47, Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Is it possible to mount a cephfs with a specific uid or gid? To make it > available to a 'non-root' user? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

backfill / recover logic (OSD included as selection criterion)
by Stefan Kooman 20 Jan '20

20 Jan '20

Hi, Is there any logic / filtering which PGs to backfill at any given time that takes into account the OSD the PG is living on? Our cluster is backfilling a complete pool now (512 PGs) and (currently) of the 7 active+remapped+backfilling there are 4 of them on the same OSD. Which stresses this OSD way more than needed. It would be nice if the selection criteria for which PG to backfill (and / or recover) from would include the OSD as selection criterion in order to spread the load across different OSDs. Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 2

Benchmark results for Seagate Exos2X14 Dual Actuator HDDs
by Paul Emmerich 16 Jan '20

16 Jan '20

Hi, we ran some benchmarks with a few samples of Seagate's new HDDs that some of you might find interesting: Blog post: https://croit.io/2020/01/06/2020-01-06-benchmark-mach2 GitHub repo with scripts and raw data: https://github.com/croit/benchmarks/tree/master/mach2-disks Tl;dr: way faster for writes, somewhat faster for reads in some scenarios Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io Looking for Ceph training? We have some free spots available https://croit.io/training/4-days-ceph-in-depth-training croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

5 6

High CPU usage by ceph-mgr in 14.2.5
by Bryan Stillwell 16 Jan '20

16 Jan '20

After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). Attaching to the thread with strace shows a lot of mmap and munmap calls. Here's the distribution after watching it for a few minutes: 48.73% - mmap 49.48% - munmap 1.75% - futex 0.05% - madvise I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this is the only one which has seen the problem (355 OSDs). Perhaps it has something to do with its size? I was suspecting it might have to do with one of the modules misbehaving, so I disabled all of them: # ceph mgr module ls | jq -r '.enabled_modules' [] But that didn't help (I restarted the mgrs after disabling the modules too). I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at me as being the cause of the problem. It only seems to affect the active mgr. If I stop the active mgr the problem moves to one of the other mgrs. Any guesses or tips on what next steps I should take to figure out what's going on? Thanks, Bryan

16 24

Uneven Node utilization
by Sasha Litvak 16 Jan '20

16 Jan '20

Hello, Cephers, I have a small 6 node cluster with 36 OSDs. When running the benchmark/torture tests I noticed that some nodes, usually storage2n6-la and also sometimes others are utilized much more. I see some osds are used 100% and load average goes up to 21 while on the others the load average is 5 - 6 and osds are within 40 - 50 - 60% of utilization. I cannot use upmap mode for balancer because I still have some client machines using hammer. I wonder if my issue is caused by compat balancing mode as compat weight shows nodes with the same number and disk size but different compat weights. If so what can I do to improve the load/disk usage distribution in the cluster? Also, my legacy client machines only need to access cephfs on the new cluster, so I wonder if keeping hammer as the oldest client version makes sense and I should change it to jewel and set crush tunables to optimal. Help is greatly appreciated, ceph df RAW STORAGE:. CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 94 TiB 88 TiB 5.9 TiB 5.9 TiB 6.29 TOTAL 94 TiB 88 TiB 5.9 TiB 5.9 TiB 6.29 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 1.6 TiB 3.77M 4.9 TiB 5.57 28 TiB cephfs_metadata 2 3.9 GiB 367.34k 4.3 GiB 0 28 TiB one 5 344 GiB 90.94k 1.0 TiB 1.20 28 TiB ceph -s cluster: id: 9b4468b7-5bf2-4964-8aec-4b2f4bee87ad health: HEALTH_OK services: mon: 3 daemons, quorum storage2n1-la,storage2n2-la,storage2n3-la (age 39h) mgr: storage2n1-la(active, since 39h), standbys: storage2n2-la, storage2n3-la mds: cephfs:1 {0=storage2n4-la=up:active} 1 up:standby-replay 1 up:standby osd: 36 osds: 36 up (since 37h), 36 in (since 10w) data: pools: 3 pools, 1664 pgs objects: 4.23M objects, 1.9 TiB usage: 5.9 TiB used, 88 TiB / 94 TiB avail pgs: 1664 active+clean io: client: 1.2 KiB/s rd, 46 KiB/s wr, 5 op/s rd, 2 op/s wr Ceph osd df looks like this ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 6 ssd 1.74609 1.00000 1.7 TiB 115 GiB 114 GiB 186 MiB 838 MiB 1.6 TiB 6.45 1.02 92 up 12 ssd 1.74609 1.00000 1.7 TiB 122 GiB 121 GiB 90 MiB 934 MiB 1.6 TiB 6.81 1.08 92 up 18 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 107 MiB 917 MiB 1.6 TiB 6.24 0.99 91 up 24 ssd 3.49219 1.00000 3.5 TiB 233 GiB 232 GiB 206 MiB 818 MiB 3.3 TiB 6.53 1.04 185 up 30 ssd 3.49219 1.00000 3.5 TiB 224 GiB 223 GiB 246 MiB 778 MiB 3.3 TiB 6.25 0.99 187 up 35 ssd 3.49219 1.00000 3.5 TiB 216 GiB 215 GiB 252 MiB 772 MiB 3.3 TiB 6.04 0.96 184 up 5 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 88 MiB 936 MiB 1.6 TiB 6.28 1.00 92 up 11 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 112 MiB 912 MiB 1.6 TiB 6.26 0.99 92 up 17 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 274 MiB 750 MiB 1.6 TiB 6.25 0.99 94 up 23 ssd 3.49219 1.00000 3.5 TiB 234 GiB 233 GiB 192 MiB 832 MiB 3.3 TiB 6.54 1.04 183 up 29 ssd 3.49219 1.00000 3.5 TiB 216 GiB 215 GiB 356 MiB 668 MiB 3.3 TiB 6.03 0.96 184 up 34 ssd 3.49219 1.00000 3.5 TiB 227 GiB 226 GiB 267 MiB 757 MiB 3.3 TiB 6.34 1.01 184 up 4 ssd 1.74609 1.00000 1.7 TiB 125 GiB 124 GiB 16 MiB 1008 MiB 1.6 TiB 7.00 1.11 94 up 10 ssd 1.74609 1.00000 1.7 TiB 108 GiB 107 GiB 163 MiB 861 MiB 1.6 TiB 6.01 0.96 93 up 16 ssd 1.74609 1.00000 1.7 TiB 107 GiB 106 GiB 163 MiB 861 MiB 1.6 TiB 6.00 0.95 94 up 22 ssd 3.49219 1.00000 3.5 TiB 221 GiB 220 GiB 385 MiB 700 MiB 3.3 TiB 6.18 0.98 187 up 28 ssd 3.49219 1.00000 3.5 TiB 223 GiB 222 GiB 257 MiB 767 MiB 3.3 TiB 6.23 0.99 186 up 33 ssd 3.49219 1.00000 3.5 TiB 241 GiB 240 GiB 233 MiB 791 MiB 3.3 TiB 6.74 1.07 185 up 1 ssd 1.74609 1.00000 1.7 TiB 103 GiB 102 GiB 240 MiB 784 MiB 1.6 TiB 5.76 0.92 93 up 7 ssd 1.74609 1.00000 1.7 TiB 117 GiB 116 GiB 70 MiB 954 MiB 1.6 TiB 6.56 1.04 91 up 13 ssd 1.74609 1.00000 1.7 TiB 126 GiB 125 GiB 76 MiB 948 MiB 1.6 TiB 7.03 1.12 95 up 19 ssd 3.49219 1.00000 3.5 TiB 230 GiB 229 GiB 307 MiB 717 MiB 3.3 TiB 6.44 1.02 186 up 25 ssd 3.49219 1.00000 3.5 TiB 220 GiB 219 GiB 309 MiB 715 MiB 3.3 TiB 6.15 0.98 185 up 31 ssd 3.49219 1.00000 3.5 TiB 223 GiB 222 GiB 205 MiB 819 MiB 3.3 TiB 6.23 0.99 186 up 0 ssd 1.74609 1.00000 1.7 TiB 116 GiB 115 GiB 151 MiB 873 MiB 1.6 TiB 6.49 1.03 93 up 3 ssd 1.74609 1.00000 1.7 TiB 121 GiB 120 GiB 89 MiB 935 MiB 1.6 TiB 6.77 1.08 91 up 9 ssd 1.74609 1.00000 1.7 TiB 104 GiB 103 GiB 183 MiB 841 MiB 1.6 TiB 5.81 0.92 93 up 15 ssd 3.49219 1.00000 3.5 TiB 222 GiB 221 GiB 205 MiB 819 MiB 3.3 TiB 6.20 0.98 185 up 21 ssd 3.49219 1.00000 3.5 TiB 213 GiB 212 GiB 312 MiB 712 MiB 3.3 TiB 5.95 0.95 182 up 27 ssd 3.49219 1.00000 3.5 TiB 221 GiB 220 GiB 219 MiB 805 MiB 3.3 TiB 6.17 0.98 185 up 2 ssd 1.74609 1.00000 1.7 TiB 104 GiB 103 GiB 116 MiB 908 MiB 1.6 TiB 5.80 0.92 92 up 8 ssd 1.74609 1.00000 1.7 TiB 111 GiB 110 GiB 118 MiB 906 MiB 1.6 TiB 6.21 0.99 91 up 14 ssd 1.74609 1.00000 1.7 TiB 106 GiB 105 GiB 192 MiB 832 MiB 1.6 TiB 5.95 0.94 92 up 20 ssd 3.49219 1.00000 3.5 TiB 226 GiB 225 GiB 196 MiB 828 MiB 3.3 TiB 6.31 1.00 185 up 26 ssd 3.49219 1.00000 3.5 TiB 232 GiB 231 GiB 231 MiB 793 MiB 3.3 TiB 6.47 1.03 184 up 32 ssd 3.49219 1.00000 3.5 TiB 226 GiB 225 GiB 229 MiB 795 MiB 3.3 TiB 6.31 1.00 184 up TOTAL 94 TiB 5.9 TiB 5.9 TiB 6.9 GiB 29 GiB 88 TiB 6.29 MIN/MAX VAR: 0.92/1.12 STDDEV: 0.31 ceph osd crush tree looks like this: ID CLASS WEIGHT (compat) TYPE NAME -1 94.28906 root default -9 15.71484 16.19038 host storage2n1-la 6 ssd 1.74609 1.77313 osd.6 12 ssd 1.74609 1.82532 osd.12 18 ssd 1.74609 2.10315 osd.18 24 ssd 3.49219 3.50087 osd.24 30 ssd 3.49219 3.01933 osd.30 35 ssd 3.49219 3.96858 osd.35 -11 15.71484 15.75711 host storage2n2-la 5 ssd 1.74609 1.84412 osd.5 11 ssd 1.74609 1.71651 osd.11 17 ssd 1.74609 1.76128 osd.17 23 ssd 3.49219 3.73497 osd.23 29 ssd 3.49219 3.27397 osd.29 34 ssd 3.49219 3.42627 osd.34 -7 15.71484 14.19093 host storage2n3-la 4 ssd 1.74609 1.66724 osd.4 10 ssd 1.74609 1.60271 osd.10 16 ssd 1.74609 1.39088 osd.16 22 ssd 3.49219 3.11852 osd.22 28 ssd 3.49219 3.04280 osd.28 33 ssd 3.49219 3.36879 osd.33 -5 15.71484 15.87343 host storage2n4-la 1 ssd 1.74609 1.92644 osd.1 7 ssd 1.74609 2.12386 osd.7 13 ssd 1.74609 1.42424 osd.13 19 ssd 3.49219 3.52307 osd.19 25 ssd 3.49219 3.55241 osd.25 31 ssd 3.49219 3.32341 osd.31 -3 15.71484 16.08948 host storage2n5-la 0 ssd 1.74609 1.97093 osd.0 3 ssd 1.74609 1.87062 osd.3 9 ssd 1.74609 1.57335 osd.9 15 ssd 3.49219 3.82397 osd.15 21 ssd 3.49219 3.59575 osd.21 27 ssd 3.49219 3.25485 osd.27 -13 15.71484 16.18745 host storage2n6-la 2 ssd 1.74609 2.18393 osd.2 8 ssd 1.74609 1.69547 osd.8 14 ssd 1.74609 1.95445 osd.14 20 ssd 3.49219 3.49811 osd.20 26 ssd 3.49219 3.42702 osd.26 32 ssd 3.49219 3.42848 osd.32 ceph balancer status { "last_optimize_duration": "0:00:00.903346", "plans": [], "mode": "crush-compat", "active": true, "optimize_result": "Unable to find further optimization, change balancer mode and retry might help", "last_optimize_started": "Thu Jan 16 05:52:53 2020" }

1 0