- ceph-users - lists.ceph.io

bluefs_buffered_io=false performance regression
by Robert Sander 11 Jan '21

11 Jan '21

Hi, bluefs_buffered_io was disabled in Ceph version 14.2.11. The cluster started last year with 14.2.5 and got upgraded over the year now running 14.2.16. The performance was OK first but got abysmal bad at the end of 2020. We checked the components and HDDs and SSDs seem to be fine. Single disk benchmarks showed performance according the specs. Today we (re-)enabled bluefs_buffered_io and restarted all OSD processes on 248 HDDs distributed over 12 nodes. Now the benchmarks are fine again: 434MB/s write instead of 60MB/s, 960MB/s read instead of 123MB/s. This setting was disabled in 14.2.11 because "in some test cases it appears to cause excessive swap utilization by the linux kernel and a large negative performance impact after several hours of run time." We have to monitor if this will happen in our cluster. Is there any other negative side effect currently known? Here are the rados bench values, first with bluefs_buffered_io=false, then with bluefs_buffered_io=true: Bench Total Total Write Object Band Stddev Max Min Average Stddev Max Min Average Stddev Max Min time writes Read size width Bandwidth IOPS Latency (s) run reads size (MB/sec) made false write 33,081 490 4194304 4194304 59,2485 71,3829 264 0 14 17,8702 66 0 1,07362 2,83017 20,71 0,0741089 false seq 15,8226 490 4194304 4194304 123,874 30 46,8659 174 0 0,51453 9,53873 0,00343417 false rand 38,2615 2131 4194304 4194304 222,782 55 109,374 415 0 0,28191 12,1039 0,00327948 true write 30,4612 3308 4194304 4194304 434,389 26,0323 480 376 108 6,50809 120 94 0,14683 0,07368 0,99791 0,0751249 true seq 13,7628 3308 4194304 4194304 961,429 240 22,544 280 184 0,06528 0,88676 0,00338191 true rand 30,1007 8247 4194304 4194304 1095,92 273 25,5066 313 213 0,05719 0,99140 0,00325295 Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 3

denied reconnect attempt for ceph fs client
by Frank Schilder 11 Jan '21

11 Jan '21

Hi all, I'm not 100% sure, but I believe that since the update from mimic-13.2.8 to mimic-13.2.10 I have a strange issue. If a ceph fs client becomes unresponsive, it is evicted, but it cannot reconnect; see ceph.log extract below. In the past, clients would retry after the blacklist period and everything continued fine. I'm wondering why the clients cannot reconnect any more. I see this now every time a client gets thrown out and didn't have this problem before. Any hints to what I might want to change are welcome, and also information what might have changed during the update (eg. is this expected or not). 2021-01-11 19:45:53.839721 [INF] denied reconnect attempt (mds is up:active) from client.30770997 192.168.56.121:0/2325067585 after 1.82224e+06 (allowed interval 45) 2021-01-11 19:45:46.713822 [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2021-01-11 19:45:45.937126 [INF] MDS health message cleared (mds.0): 1 slow requests are blocked > 30 secs 2021-01-11 19:45:39.527180 [INF] Evicting (and blacklisting) client session 30770997 (192.168.56.121:0/2325067585) 2021-01-11 19:45:39.527168 [WRN] evicting unresponsive client HOSTNAME:CLIENT_NAME (30770997), after 62.735 seconds 2021-01-11 19:45:39.522141 [WRN] 1 slow requests, 0 included below; oldest blocked for > 50.371751 secs 2021-01-11 19:45:34.522085 [WRN] 1 slow requests, 0 included below; oldest blocked for > 45.371685 secs 2021-01-11 19:45:29.521991 [WRN] 1 slow requests, 0 included below; oldest blocked for > 40.371604 secs 2021-01-11 19:45:24.521907 [WRN] 1 slow requests, 0 included below; oldest blocked for > 35.371520 secs 2021-01-11 19:45:19.521895 [WRN] slow request 30.371469 seconds old, received at 2021-01-11 19:44:49.150361: client_request(client.30771333:10419033 getattr pAsLsXsFs #0x10014446b0f 2021-01-11 19:44:49.145012 caller_uid=257062, caller_gid=257062{}) currently failed to rdlock, waiting Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

DocuBetter Meeting This Week -- 13 Jan 2021 1730 UTC
by John Zachary Dover 11 Jan '21

11 Jan '21

Unless an unforeseen crisis arises, the DocuBetter meetings for the next two months will focus on ensuring that we have a smooth and easy-to-understand docs suite for the release of Pacific. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

1 0

[cephadm] Point release minor updates block themselves infinitely
by Paul Browne 11 Jan '21

11 Jan '21

Hello all, I've been having some real troubles in getting cephadm to apply some very minor point release updates cleanly, twice now applying the point update of 15.2.6 -> 15.2.7 and 15.2.7 to 15.2.8 has gotten blocked somewhere and ended up making no progress, requiring digging deep into internals to unblock things. In the most recent attempt of 15.2.7 -> 15.2.8, the Orchestrator cleanly replaced Mon and MGR containers in the first steps, but when it came to replacing Crash daemon containers the running 15.2.7 Crash container was purged but container update operations then seem to get blocked on trying to start it again on the older image, leading to an infinite loop of Podman trying to start a non-existent container in logging; https://pastebin.com/9zdMs1XU Forcing an `ceph orch daemon rm` of the Crash daemon affected for the host just repeats the loop again. I'd then tried removing the Crash service and all daemons through the Orchestrator API next. This purged all running Crash containers from all hosts, and then re-applyed a service spec to restart them, hopefully on the new image. The Orchestrator removal of the Crash containers seems to have left container state dangling on hosts however, as now we see the same issue of Crash containers not starting on *every* host in the cluster due to left-over container state ; https://pastebin.com/tjaegxqg At this point I'm not certain if Podman (v1.6.4 EPEL, CentOS7.9) or Orchestrator is to blame for leaving this state dangling and blocking new container creation, but it's proving a real problem in applying even simple minor version point updates. Has anyone else been seeing similar behaviour in applying minor version updates via cephadm+Orchestrator? Are there any good workarounds to clean up the dangling container state? -- ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk Tel: 0044-1223-746548 *******************

1 1

osd gradual reweight question
by mj 11 Jan '21

11 Jan '21

Hi, We are replacing HDD with SSD, and we first (gradually) drain (reweight) the HDDs with 0.5 steps until 0 = empty. Works perfectly. Then (just for kicks) I tried reducing HDD weight from 3.6 to 0 in one large step. That seemed to have had more impact on the cluster, and we even noticed some OSD's temporarily go down after a few minutes. It all worked out, but the impact seemed much larger. We never had OSDs go down when gradually reducing the weight step by step. This surprised us. Is it expected that the impact of a sudden reweight from 3.6 to 0 is bigger than a gradual step-by-step decrease? I would assume the impact to be similar, only the time it takes to reach HEALTH_OK to be longer. Thanks, MJ

3 3

performance impact by pool deletion?
by Scheurer François 11 Jan '21

11 Jan '21

Hi everybody Does somebody had experience with important performance degradations during a pool deletion? We are asking because we are going to delete a 370 TiB with 120 M objects and have never done this in the past. The pool is using erasure coding 8+2 on nvme ssd's with rocksdb/wal on nvme optane disks. Openstack VM's are running on the other rbd pools. Thank you in advance for your feedback! Cheers Francois PS: this is no option, as it take about 100 year to complete ;-) : rados -p ch-zh1-az1.rgw.buckets.data ls | while read i; do rados -p ch-zh1-az1.rgw.buckets.data rm "$i"; done -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

4 5

Snaptrim making cluster unusable
by Pascal Ehlert 10 Jan '21

10 Jan '21

Hi all, We are running a small cluster with three nodes and 6-8 OSDs each. The OSDs are SSDs with sizes from 2 to 4 TB. Crush map is configured so all data is replicated to each node. The Ceph version is Ceph 15.2.6. Today I deleted 4 Snapshots of the same two 400GB and 500GB rbd volumes. Shortly after issuing the delete, I noticed the cluster became unresponsive to an extend where almost all our services went down due high IO latency. After a while, I noticed about 20 active snaptrim tasks + another 200 or so snaptrim_wait. I tried setting osd_snap_trim_sleep to 3, osd_pg_max_concurrent_snap_trims to 1 rbd_balance_snap_reads to true, rbd_localize_snap_reads to true Still the only way to make the cluster responsive again was to set osd_pg_max_concurrent_snap_trims to 0 and thus disable snaptrimming. I tried a few other options, but whenever snaptrims are running for a significant number of PGs, the cluster becomes completely unusable. Are there any other options to throttle snaptrimming for that I haven't tried, yet? Thank you, Pascal

3 6

Nautils BlueStore OSDs will not start
by William Law 08 Jan '21

08 Jan '21

We have several OSDs that are crashing on start. We are running nautilus 14.2.16; here is the relevant bit of the log: -10> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read left 0xf1000 len 0x1000 -9> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read got 4096 -8> 2021-01-08 14:52:38.800 7feec5f27c00 10 bluefs _replay 0x10f000: txn(seq 5972194 len 0x27 crc 0x9783dfc6) -7> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _replay 0x10f000: op_file_update file(ino 68757 size 0x6041f mtime 2021-01-07 21:25:57.793664 allocated 100000 extents [1:0x17e00000~100000]) -6> 2021-01-08 14:52:38.800 7feec5f27c00 10 bluefs _read h 0x55abfb20a3c0 0x110000~1000 from file(ino 1 size 0x110000 mtime 0.000000 allocated 500000 extents [1:0x481700000~100000,1:0x7ad00000~400000]) -5> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read left 0xf0000 len 0x1000 -4> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _read got 4096 -3> 2021-01-08 14:52:38.800 7feec5f27c00 10 bluefs _replay 0x110000: txn(seq 5972195 len 0x116 crc 0xec6cec7) -2> 2021-01-08 14:52:38.800 7feec5f27c00 20 bluefs _replay 0x110000: op_dir_link db/109668.log to 68759 -1> 2021-01-08 14:52:38.804 7feec5f27c00 -1 /build/ceph-14.2.16/src/os/bluestore/BlueFS.cc<http://BlueFS.cc>: In function 'int BlueFS::_replay(bool, bool)' thread 7feec5f27c00 time 2021-01-08 14:52:38.802560 /build/ceph-14.2.16/src/os/bluestore/BlueFS.cc<http://BlueFS.cc>: 1029: FAILED ceph_assert(file->fnode.ino) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55abefa51fba] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55abefa52195] 3: (BlueFS::_replay(bool, bool)+0x4fa5) [0x55abf006f8f5] 4: (BlueFS::mount()+0x229) [0x55abf006fd79] 5: (BlueStore::_open_bluefs(bool)+0x78) [0x55abeff57958] 6: (BlueStore::_open_db(bool, bool, bool)+0x8a3) [0x55abeff58e63] 7: (BlueStore::_open_db_and_around(bool)+0x44) [0x55abeff6a1a4] 8: (BlueStore::_mount(bool, bool)+0x584) [0x55abeffc0b64] 9: (OSD::init()+0x3f3) [0x55abefb01db3] 10: (main()+0x5214) [0x55abefa5acf4] 11: (__libc_start_main()+0xe7) [0x7feec279cbf7] 12: (_start()+0x2a) [0x55abefa8c72a] 0> 2021-01-08 14:52:38.808 7feec5f27c00 -1 *** Caught signal (Aborted) ** in thread 7feec5f27c00 thread_name:ceph-osd ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) It seems like a version of this: https://tracker.ceph.com/issues/45519 and maybe this https://tracker.ceph.com/issues/21087I haven't been able to get it to start with the stupid allocator. Here is the ceph.conf I've been using to try to get it to start; a thread seemed to indicate that increasing the bluefs_max_log_runway would help. [global] fsid =<redacted> mon initial members = <redacted> public_network = 10.210.20.0/22 mon_host = 10.210.20.21,10.210.20.22,10.210.20.23 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx #kernel doesn't support all features, so disable this #rbd default features = 3 [osd] #set snap trim priority to lowest, 1 osd_snap_trim_priority = 1 osd_recovery_op_priority = 8 osd-max-backfills = 16 osd-max-backfills = 1 #keep allocator commented out normally; defaults to bitmap but #may need stupid #inspired by https://tracker.ceph.com/issues/45519 bluestore_allocator = stupid bluefs_allocator = stupid debug_bluefs = 20/20 #set to 3x default 4194304 bluefs_max_log_runway = 12582912 The cluster is having issues and it is urgent we get these OSDs up. The DB is on a shared NVMe device (81G) and the disk is a 2.2TB 2.5 inch enterprise disk. I'll be very grateful for any assistance. Best, Will

1 0

Ceph orch syntax to create OSD on a partition?
by Marc Spencer 08 Jan '21

08 Jan '21

All, I acknowledge it’s not a fantastic idea, but I wish to create an OSD on a partition rather than an entire device. I can’t seem to find the answer in the documentation. Is there a way using ceph orch daemon add osd to identify targets other than a whole device? Thanks in advance. - m

2 1

Larger number of OSDs, cheroot, cherrypy, limits + containers == broken
by David Orman 07 Jan '21

07 Jan '21

Hi, We have a ceph 15.2.7 deployment using cephadm under podman w/ systemd. We've run into what we believe is: https://github.com/ceph/ceph-container/issues/1748 https://tracker.ceph.com/issues/47875 In our case, eventually the mgr container stops emitting output/logging. We are polling with external prometheus clusters, which is likely what triggers the issue, as it appears some amount of time after the container is spawned. Unfortunately, setting limits in the systemd service file for the mgr service on the host OS doesn't work, nor does modifying the unit.run file which is used to start the container under podman to include the --ulimit settings as suggested. Looking inside the container: lib/systemd/system/ceph-mgr@.service:LimitNOFILE=1048576 This prevents us from deploying medium to large ceph clusters, so I would argue it's a high priority bug that should not be closed, unless there is a workaround that works until EPEL 8 contains the fixed version of cheroot and the ceph containers include it. My understanding is this was fixed in cheroot 8.4.0: https://github.com/cherrypy/cheroot/issues/249 https://github.com/cherrypy/cheroot/pull/301 Thank you in advance for any suggestions, David

4 10