- ceph-users - lists.ceph.io

bluestore_default_buffered_write = true
by Adam Koczarski 21 Jul '20

21 Jul '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

2 1

ceph osd log -> set_numa_affinity unable to identify public interface
by EDH - Manuel Rios 21 Jul '20

21 Jul '20

Hi , Today checking the osd logs at boot after upgrade to 14.2.10 we found that: set_numa_affinity unable to identify public interface 'p3p1.4094' numa node: (2) No such file or directory "2020-07-20 20:41:41.134 7f2cd15ca700 -1 osd.12 1120769 set_numa_affinity unable to identify public interface 'p3p1.4094' numa node: (2) No such file or directory" Sounds weird it can find numa based in the p3p1 but no in p3p1.<vlan-id> Can we ignore that "warning" or just need to tune something more? Regards Manuel

1 0

Degradation of write-performance after upgrading to Octopus
by Thomas Gradisnik 20 Jul '20

20 Jul '20

We have deployed a small test cluster consisting of three nodes. Each node is running a mon/mgr and two osds (Samsung PM983 3,84TB NVMe split into two partitions), so six osds in total. We started with Ceph 14.2.7 some weeks ago (upgraded to 14.2.9 later) and ran different tests using fio against some rbd volumes in order to get an overview what performance we could expect. The configuration is unchanged compared to the defaults, we only set several debugging options to 0/0. Yesterday we upgraded the whole cluster following the upgrade guidelines to Ceph 15.2.3, which worked without any problems so far. Nevertheless when running the same tests as before with Ceph 14.2.9, we are seeing some clear degradations in write-performance (beside some performance improvements, which shall also be mentioned). Here the results of concern (each with the relevant fio settings used): Test "read-latency-max" (rw=randread, iodepth=64, bs=4k) read_iops: 32500 -> 87000 Test "write-latency-max" (rw=randwrite, iodepth=64, bs=4k) write_iops: 22500 -> 11500 Test "write-throughput-iops-max" (rw=write, iodepth=64, bs=4k) write_iops: 7000 -> 14000 Test "usecase1" (rw=randrw, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/,4k/50:8k/20:16k/20:32k/5:64k/2:128k/:256k/, rwmixread=1, rate_process=poisson, iodepth=64) write_iops: 21000 -> 8500 Test "usecase1-readonly" (rw=randread, bssplit=4k/40:8k/5:16k/20:32k/5:64k/10:128k/10:256k/, rate_process=poisson, iodepth=64) read_iops: 28000 -> 58000 The last two tests represent a typical use case on our systems. Therefore we are especially concerned by the drop in performance from 21000 w/ops to 8500 w/ops (about 60%) after upgrading to Ceph 15.2.3. We ran all tests several times, the values are averaged over all iterations and fairly consistent and reproducible. We even tried wiping the whole cluster, downgrading to Ceph 14.2.9 again, setting up a new cluster/pool, running the tests and upgrading to Ceph 15.2.3 again. The tests have been performed on one of the three cluster nodes using a 50G rbd volume, which had been prefilled with random data before each test-run. Have any changes been introduced with Octopus that could explain the observed changes in performance? What we already tried: - Disabling rbd cache - Reverting rbc cache policy to writeback (default in 14.2) - Setting rbd io scheduler to none - Deploying a fresh cluster starting with Ceph 15.2.3 Kernel is 5.4.38 … I don't know if some other system specs would be helpful besides the already mentioned (since we are talking about a relative change in performance after upgrading Ceph without any further changes) - if so, please let us know.

6 10

ceph/rados performace sync vs async
by Daniel Mezentsev 20 Jul '20

20 Jul '20

Hi All, I started a small project related to metrics collection and processing, Ceph was chosen as a storage backend. Decided to use rados directly, to avoid any additional layers. I got a very simple client - it works fine, but performance is very low. Can't get more than 30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be mentioned about the client - I'm using sbcl (yep lisp). Call to rados API is just cffi call. Did try in async mode. Wow ! Saturated network bandwidth for large objects (4Mb and bigger), for small objects - saturated OSD IOPS ~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty reasonable. Bottom line - issue not with the lisp client - im getting close to C performance, difference is sync vs async IO. Why it's so big - sync operations are approx 2-3 times slower then async . Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

4 6

HP printer offline
by masonlava77＠gmail.com 20 Jul '20

20 Jul '20

When I make my hard efforts to print the documents through my HP printer, suddenly my HP printer goes offline mode. I am applying an appropriate command to my HP printer, I am facing HP printer offline problem. This offline error is an annoying issue for me, so I am not able to work on my HP printer. I am trying to find out the major reasons of HP printer offline error, but I don’t get the success to find out actual reasons. As per my technical skills, I have applied my technical knowledge to sort out HP printer offline issue. So anyone can suggest the smart methods to rectify this offline error. https://www.hpprintersupportpro.com/blog/stop-my-hp-printer-from-going-offl…

1 0

HP printer offline
by masonlava77＠gmail.com 20 Jul '20

20 Jul '20

When I make my hard efforts to print the documents through my HP printer, suddenly my HP printer goes offline mode. I am applying an appropriate command to my HP printer, I am facing HP printer offline problem. This offline error is an annoying issue for me, so I am not able to work on my HP printer. I am trying to find out the major reasons of HP printer offline error, but I don’t get the success to find out actual reasons. As per my technical skills, I have applied my technical knowledge to sort out HP printer offline issue. So anyone can suggest the smart methods to rectify this offline error. https://www.hpprintersupportpro.com/blog/stop-my-hp-printer-from-going-offl…

1 0

Cache Tier OSDs full and near full - not flushing and evicting
by Priya Sehgal 20 Jul '20

20 Jul '20

Hi, We have a large production cluster with Writeback Cache Tier. Recently, we observed that some of the OSDs were near full and got full from the Cache Tier and the cluster is in error state. The target_max_bytes was not set correctly and hence I think the flush and eviction never happened. I wish to disable caching and remove the cache tier, but before that wish to flush all the contents of the cache. I have set the cache mode to "forward" as suggested here: https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/1.2.3/ht…. But, rados df still shows same usage of cache tier. It's not automatically flushing and evicting. I tried "cache-try-flush", but this command gets blocked for some reason for many objects (although this is supposed to be non-blocking). How do I make sure all the objects are flushed to backing tier and objects evicted so that I can safely deprecate/remove the cache tier which is currently having many full and near full OSDs (in cache tier). -- Regards, Priya

1 0

ceph OSD node optimised sysctl configuration
by Jeremi Avenant 20 Jul '20

20 Jul '20

Hi All Is there perhaps any updated documentation about ceph OSD node optimised sysctl configuration? I'm seeing a lot of these: $ netstat -s ... 4955341 packets pruned from receive queue because of socket buffer overrun ... 5866 times the listen queue of a socket overflowed ... TCPBacklogDrop: 14601 OS: Ubuntu 16 HWE Ceph Luminous : 12.2.13 Ceph-ansible: 3.2.20 + containerised Thanks in advance. -- *Jeremi-Ernst Avenant, Mr.* Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

1 0

"ceph daemon osd.x ops" shows different number from "ceph osd status <bucket>"
by rainning 20 Jul '20

20 Jul '20

"ceph daemon osd.x ops" shows ops currently in flight, the number is different from "ceph osd status <bucket>". Is the number of ops in "ceph osd status <bucket>" also the ops currently in flight? root@osd021:~# ceph daemon osd.636 ops {     "ops": [],     "num_ops": 0 } root@osd021:~# ceph osd status osd021 +-----+--------+-------+-------+--------+---------+--------+---------+-----------+ |  id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   | +-----+--------+-------+-------+--------+---------+--------+---------+-----------+ |  37 | osd021 |  542G | 1134G |   22   |   225k  |   12   |   274k  | exists,up | |  91 | osd021 |  567G | 1109G |   38   |   281k  |   12   |  1006k  | exists,up | | 147 | osd021 |  551G | 1125G |   32   |   307k  |    4   |   247k  | exists,up | | 201 | osd021 |  549G | 1127G |   82   |  1505k  |    5   |  42.4k  | exists,up | | 264 | osd021 | 1503G | 2222G |    3   |  55.2k  |    4   |  12.2k  | exists,up | | 317 | osd021 | 1537G | 2188G |   13   |   138k  |    1   |  6190   | exists,up | | 375 | osd021 | 1369G | 2356G |    1   |   834k  |    5   |  67.4k  | exists,up | | 430 | osd021 | 1424G | 2301G |    1   |  8192   |    6   |  1661k  | exists,up | | 480 | osd021 | 1435G | 2289G |    3   |  16.6k  |    2   |  42.4k  | exists,up | | 535 | osd021 | 1494G | 2231G |    2   |  12.0k  |    1   |  27.8k  | exists,up | | 584 | osd021 | 1484G | 2241G |    1   |  5529   |    1   |  12.7k  | exists,up | | 636 | osd021 | 1394G | 2331G |    2   |   850k  |  406   |  78.1M  | exists,up | +-----+--------+-------+-------+--------+---------+--------+---------+-----------+

1 0

Ceph config changed on ansible deployed cluster
by Szabo, Istvan (Agoda) 19 Jul '20

19 Jul '20

Hi, How is it possible to change config on an ansible deployed cluster due to the missing ceph-deploy. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2