- ceph-users - lists.ceph.io

Limited performance
by Fabian Zimmermann 25 Feb '20

25 Feb '20

Hi, we currently creating a new cluster. This cluster is (as far as we can tell) an config-copy (ansible) of our existing cluster, just 5 years later - with new hardware (nvme instead of ssd, bigger disks, ...) The setup: * NVMe for Journals and "Cache"-Pool * HDD with NVMe Journals for "Data"-Pool * Cache-Pool as writeback-Tier on Data-Pool * We are using 12.2.13 without bluestore. If we run an rados benchmark against this pool, everything seems fine, but as soon as we start a fio-benchmark -<- [global] ioengine=rbd clientname=cinder pool=cinder rbdname=fio_test rw=write bs=4M [rbd_iodepth32] iodepth=32 ->- after some seconds the bandwidth drops to <15 MB/s and our hdd-disks are doing more IOs than our Journal-Disks. We also unconfigured the caching completely, but the issue remains. The output of "ceph osd pool stats" shows ~100 op/s, but our disks are doing: -<- Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 0.00 0.00 278.50 0.00 34.07 250.51 0.14 0.50 0.00 0.50 0.03 0.80 nvme1n1 0.00 0.00 0.00 64.00 0.00 7.77 248.50 0.01 0.22 0.00 0.22 0.03 0.20 sda 0.00 1.50 0.00 557.00 0.00 29.49 108.45 180.57 160.59 0.00 160.59 1.80 100.00 sdb 0.00 42.00 0.00 592.00 0.00 28.21 97.60 176.51 1105.79 0.00 1105.79 1.69 100.00 sdc 0.00 14.50 0.00 528.50 0.00 27.95 108.31 183.02 179.47 0.00 179.47 1.89 100.00 sde 0.00 134.50 0.00 223.50 0.00 14.05 128.72 17.38 60.05 0.00 60.05 0.89 20.00 sdg 0.00 76.00 0.00 492.00 0.00 26.32 109.54 191.81 1474.96 0.00 1474.96 2.03 100.00 sdf 0.00 0.00 0.00 491.50 0.00 26.76 111.49 176.55 326.05 0.00 326.05 2.03 100.00 sdh 0.00 0.00 0.00 548.50 0.00 26.71 99.75 204.39 327.57 0.00 327.57 1.82 100.00 sdi 0.00 112.00 0.00 526.00 0.00 23.15 90.14 158.32 1325.61 0.00 1325.61 1.90 100.00 sdj 0.00 12.00 0.00 641.00 0.00 34.78 111.13 185.51 278.29 0.00 278.29 1.56 100.00 sdk 0.00 23.50 0.00 399.50 0.00 20.38 104.46 166.77 461.67 0.00 461.67 2.50 100.00 sdl 0.00 267.00 0.00 498.50 0.00 34.46 141.58 200.37 490.80 0.00 490.80 2.01 100.00 ->- Any hints how to debug the issue? Thanks a lot, Fabian

3 2

Unable to increase PG numbers
by Andres Rojas Guerrero 25 Feb '20

25 Feb '20

Hi, I have a Nautilus installation version 14.2.1 with a very unbalanced cephfs pool, I have 430 osd in the cluster but this pool only have 8 PG and PGP and 118 TB used : # ceph -s cluster: id: a2269da7-e399-484a-b6ae-4ee1a31a4154 health: HEALTH_WARN 1 nearfull osd(s) 2 pool(s) nearfull services: mon: 3 daemons, quorum mon21,mon22,mon23 (age 7M) mgr: mon23(active, since 8M), standbys: mon22, mon21 mds: cephfs:2 {0=mon21=up:active,1=mon22=up:active} 1 up:standby osd: 430 osds: 430 up, 430 in data: pools: 2 pools, 16 pgs objects: 10.07M objects, 38 TiB usage: 118 TiB used, 4.5 PiB / 4.6 PiB avail pgs: 15 active+clean 1 active+clean+scrubbing+deep # ceph osd pool get cephfs_data pg_num pg_num: 8 Due to this bad configuration I have this warning message: # ceph status cluster: id: a2269da7-e399-484a-b6ae-4ee1a31a4154 health: HEALTH_WARN 1 nearfull osd(s) 2 pool(s) nearfull I've discover that some osd are full: # ceph osd status | 113 | osd23 | 9824G | 1351G | 0 | 0 | 0 | 0 | exists,nearfull,up | I've tried to reweight this osd: ceph osd reweight osd.113 0.9 But the process of reweight doesn't start. Otherwise I've tried to increase the PG and PGP numbers but it' doesn't work. # ceph osd pool set cephfs_data pg_num 128 set pool 1 pg_num to 128 # ceph osd pool get cephfs_data pg_num pg_num: 8 What could be the reason for this problem?

3 5

Migrating data to a more efficient EC pool
by Vladimir Brik 25 Feb '20

25 Feb '20

Hello I have ~300TB of data in default.rgw.buckets.data k2m2 pool and I would like to move it to a new k5m2 pool. I found instructions using cache tiering[1], but they come with a vague scary warning, and it looks like EC-EC may not even be possible [2] (is it still the case?). Can anybody recommend a safe procedure to copy an EC pool's data to another pool with a more efficient erasure coding? Perhaps there is a tool out there that could do it? A few days of downtime would be tolerable, if it will simplify things. Also, I have enough free space to temporarily store the k2m2 data in a replicated pool (if EC-EC tiering is not possible, but EC-replicated and replicated-EC tiering is possible). Is there a tool or some efficient way to verify that the content of two pools is the same? Thanks, Vlad [1] https://ceph.io/geen-categorie/ceph-pool-migration/ [2] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-February/016109.ht…

2 1

Changing allocation size
by Kristof Coucke 25 Feb '20

25 Feb '20

Hi all, A while back, I indicated we had an issue with our cluster filling up too fast. After checking everything, we've concluded this was because we had a lot of small files and the allocation size on the bluestore was too high (64kb). We are now recreating the OSD's (2 disk at the same time) but, this will take a very long time as we're dealing with 130 OSDs. The current process we're following is removing 2 osd's and recreating them. We're using erasure coding (6 + 3). Has anyone some advice on how we can move forward with this? We've already increased some parameters to speed up recovery, but even then, it would still cost us too much time. If we could recreate them faster, that would be great... Or adapt the allocation on the fly? Any suggestions are welcome... Thank you, Kristof.

2 1

unscheduled mds failovers
by danjou.philippe＠yahoo.de 25 Feb '20

25 Feb '20

Hi, we're seeing active mds failover to standbys every few weeks causing few minutes of cephfs downtime. It's not crashing, all the log says is. 2020-02-25 08:30:53.313 7f9a457ae700 1 mds.m2-1045557 Updating MDS map to version 10132 from mon.1 2020-02-25 08:30:53.313 7f9a457ae700 1 mds.m2-1045557 Map has assigned me to become a standby Anyone got any idea why that is? It's not so nice that ceph does it by itself. Thanks

1 0

Ceph @ SoCal Linux Expo
by Gregory Farnum 24 Feb '20

24 Feb '20

Hey all, we're excited to be returning properly to SCaLE in Pasadena[1] this year (March 5-8) with a Thursday Birds-of-a-Feather session[2] and a booth in the expo hall. Please come by if you're attending the conference or are in the area to get face time with other area users and Ceph developers. :) Also, I got drafted into organizing this so if you'd be willing to help man the booth in exchange for an Expo pass, shoot me an email! I think I've got 3 spots left. -Greg [1]: https://www.socallinuxexpo.org/scale/18x [2]: https://www.socallinuxexpo.org/scale/18x/presentations/ceph-storage

1 0

ceph-mon using 100% CPU after upgrade to 14.2.5
by Bryan Stillwell 24 Feb '20

24 Feb '20

On our test cluster after upgrading to 14.2.5 I'm having problems with the mons pegging a CPU core while moving data around. I'm currently converting the OSDs from FileStore to BlueStore by marking the OSDs out in multiple nodes, destroying the OSDs, and then recreating them with ceph-volume lvm batch. This seems too get the ceph-mon process into a state where it pegs a CPU core on one of the mons: 1764450 ceph 20 0 4802412 2.1g 16980 S 100.0 28.1 4:54.72 ceph-mon Has anyone else run into this with 14.2.5 yet? I didn't see this problem while the cluster was running 14.2.4. Thanks, Bryan

5 8

RGW do not show up in 'ceph status'
by Andreas Haupt 24 Feb '20

24 Feb '20

Dear all, we recently added two additional RGWs to our CEPH cluster (version 14.2.7). They work flawlessly, however they do not show up in 'ceph status': [cephmon1] /root # ceph -s | grep -A 6 services services: mon: 3 daemons, quorum cephmon1,cephmon2,cephmon3 (age 14h) mgr: cephmon1(active, since 14h), standbys: cephmon2, cephmon3 mds: cephfs:1 {0=cephmon1=up:active} 2 up:standby osd: 168 osds: 168 up (since 2w), 168 in (since 6w) rgw: 1 daemon active (ceph-s3) As you can see, only the first, old RGW (ceph-s3) is listed. Is there any place where the RGWs need to get "announced"? Any idea, how to debug this? Thanks, Andreas -- | Andreas Haupt | E-Mail: andreas.haupt(a)desy.de | DESY Zeuthen | WWW: http://www-zeuthen.desy.de/~ahaupt | Platanenallee 6 | Phone: +49/33762/7-7359 | D-15738 Zeuthen | Fax: +49/33762/7-7216

2 3

One PG is stuck and reading is not possible
by mikko.lampikoski＠doop.fi 24 Feb '20

24 Feb '20

ceph version 12.2.13 luminous (stable) My whole ceph cluster went to kind of read only state. Ceph status showed that client reads is 0 op/s for whole cluster. There was normal amount of writes going on. I checked health and it said: # ceph health detail HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg peering PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg peering pg 26.13b is stuck peering for 25523.506788, current state peering, last acting [2,0,33] All osds showed to be up and all monitors are good. All pools are 3/2 (size/min) and space usage ~30%. I fixed this by restarting forst osd.2 (nothing happened) and then restarted osd.0. After that everyting went back to normal. So what can cause "stuck peering" and how can i prevent this event from happening again?

1 0

Fwd: MDS Daemon Damaged
by Ben 24 Feb '20

24 Feb '20

I have a 3 node ceph cluster for my house that I have been using for a few years now without issue. Each node is a MON, MGR, and MDS, and has 2-3 OSDs on them. It has, however been slow. I decided to finally move the bluestore DBs to SSDs. I did one OSD as a test case to make sure everything was going to go OK. I deleted the OSD, then created a new OSD using the ceph-deploy tool and pointed the DB at a LVM partition on a SSD. Everything went OK, and recovery started. Later in the day I noticed that my MDS daemon is damaged (PGs are still recovering). I've tried the cephfs-journal-tool --rank=cephfs:all journal export backup.bin command, but it gave me: 2020-02-23 17:50:03.589 7f7d8b225740 -1 Missing object 200.00c30b6d 2020-02-23 17:50:07.919 7f7d8b225740 -1 Bad entry start ptr (0x30c2dbb92003) at 0x30c2d3a125ea (both lines have several repeats) and will not complete. Looking at the log file of the mds that was active at the time shows: 2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro) _finish_read got error -2 2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro) _finish_read got error -2 2020-02-23 17:13:09.091 7fad40029700 0 mds.0.journaler.mdlog(ro) _finish_read got error -2 2020-02-23 17:13:09.091 7fad3e826700 0 mds.0.log _replay journaler got error -2, aborting 2020-02-23 17:13:09.091 7fad3e826700 -1 log_channel(cluster) log [ERR] : missing journal object One other thing that happened about the same time, I noticed I was having memory pressure on all the nodes with only 200MB of free ram. I've tweaked the bluestore osd_memory_target to try to help that not happen again. Even so, I'm a bit confused how that could cause catastrophic failure, as I had 2 other MDSes on standby. Any help would be appreciated.

1 0