July 2019 - ceph-users - lists.ceph.io

MDS failing under load with large cache sizes
by Janek Bevendorff 07 Jan '20

07 Jan '20

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 33

Help needed pls (rbd cache)
by Muhammad Junaid 02 Aug '19

02 Aug '19

Hi there, Sorry for asking a question, which may be of very basic nature and asked many times before. But much Google search can not satisfy me. The question is about RBD Cache in write-back mode using KVM/libvirt. If we enable this, it uses local KVM Host's RAM as cache for VM's write requests. And KVM Host immediately responds to VM's OS that data has been written to Disk (Actually it is still not on OSD's yet). Then how can be it power failure safe? Is my understanding correct? If not, pls correct. This is very important for me. Thank you very much in advance. Best regards. Muhammad Junaid

2 2

Ceph rbd cannot delete after I reset ceph cluster
by xu xingci 31 Jul '19

31 Jul '19

Hi guys: I had setup ceph cluster and mount rbd on one machine. I delete ceph cluster and reinstall follow the manual. but I still have rbd device mount on my machine. I can not access mount point. This is my detail info, I want to delete all old rbd device, what should I do? node1 $> rbd device list id pool namespace image snap device 0 rbd foo - /dev/rbd0 1 kube kubernetes-dynamic-pvc-1cc43c5b-ade1-11e9-9a92-863e3c12afd1 - /dev/rbd1 node1 $> df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/rootvg-lv_root 10995712 2528388 8467324 23% / devtmpfs 2012656 0 2012656 0% /dev tmpfs 2023588 0 2023588 0% /dev/shm tmpfs 2023588 207340 1816248 11% /run tmpfs 2023588 0 2023588 0% /sys/fs/cgroup /dev/sda1 520868 116936 403932 23% /boot /dev/mapper/rootvg-lv_var 5232640 3226816 2005824 62% /var /dev/mapper/rootvg-lv_tmp 5232640 33060 5199580 1% /tmp /dev/rbd0 3997376 16392 3754888 1% /mnt tmpfs 404720 0 404720 0% /run/user/1001 node1 $> rbd trush list rbd: error opening default pool 'rbd' Ensure that the default pool has been created or specify an alternate pool name. node1 $> rbd info rbd/foo rbd: error opening default pool 'rbd' Ensure that the default pool has been created or specify an alternate pool name. [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-anima…]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> 无病毒。www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…>

1 0

Help pls (RBD Cache)
by Muhammad Junaid 30 Jul '19

30 Jul '19

Hi there, Sorry for asking a question, which may be of very basic nature and asked many times before. But much Google search can not satisfy me. The question is about RBD Cache in write-back mode using KVM/libvirt. If we enable this, it uses local KVM Host's RAM as cache for VM's write requests. And KVM Host immediately responds to VM's OS that data has been written to Disk (Actually it is still not on OSD's yet). Then how can be it power failure safe? Is my understanding correct? If not, pls correct. This is very important for me. Thank you very much in advance. Best regards. Muhammad Junaid

1 0

Re: Fwd: [lca-announce] linux.conf.au 2020 - Call for Sessions and Miniconfs now open!
by Tim Serong 25 Jul '19

25 Jul '19

Hi All, Just a reminder, there's only a few days left to submit talks for this most excellent conference; the CFP is open until Sunday 28 July Anywhere on Earth. (I've submitted a Data Storage miniconf day, fingers crossed...) Regards, Tim On 6/26/19 2:09 PM, Tim Serong wrote: > Here we go again! As usual the conference theme is intended to > inspire, not to restrict; talks on any topic in the world of free and > open source software, hardware, etc. are most welcome, and Ceph talks > definitely fit. > > I've added this to https://pad.ceph.com/p/cfp-coordination as well. > > -------- Forwarded Message -------- > Subject: [lca-announce] linux.conf.au 2020 - Call for Sessions and > Miniconfs now open! > Date: Tue, 25 Jun 2019 21:19:43 +1000 > From: linux.conf.au Announcements <lca-announce(a)lists.linux.org.au> > Reply-To: lca-announce(a)lists.linux.org.au > To: lca-announce(a)lists.linux.org.au > > > The linux.conf.au 2020 organising team is excited to announce that the > linux.conf.au 2020 Call for Sessions and Call for Miniconfs are now open! > These will stay open from now until Sunday 28 July Anywhere on Earth > (AoE) (https://en.wikipedia.org/wiki/Anywhere_on_Earth) > > Our theme for linux.conf.au 2020 is "Who's Watching", focusing on > security, privacy and ethics. > As big data and IoT-connected devices become more pervasive, it's no > surprise that we're more concerned about privacy and security than ever > before. > We've set our sights on how open source could play a role in maximising > security and protecting our privacy in times of uncertainty. > With the concept of privacy continuing to blur, open source could be the > solution to give us '2020 vision'. > > Call for Sessions > > Would you like to talk in the main conference of linux.conf.au 2020? > The main conference runs from Wednesday to Friday, with multiple streams > catering for a wide range of interest areas. > We welcome you to submit a session > (https://linux.conf.au/programme/sessions/) proposal for either a talk > or tutorial now. > > Call for Miniconfs > > Miniconfs are dedicated day-long streams focusing on single topics, > creating a more immersive experience for delegates than a session. > Miniconfs are run on the first two days of the conference before the > main conference commences on Wednesday. > If you would like to organise a miniconf > (https://linux.conf.au/programme/miniconfs/) at linux.conf.au, we want > to hear from you. > > Have we got you interested? > > You can find out how to submit your session or miniconf proposals at > https://linux.conf.au/programme/proposals/. > If you have any other questions you can contact us via email at > contact(a)lca2020.linux.org.au. > > We are looking forward to reading your submissions. > > linux.conf.au 2020 Organising Team > > > --- > Read this online at > https://lca2020.linux.org.au/news/call-for-sessions-miniconfs-now-open/ > _______________________________________________ > lca-announce mailing list > lca-announce(a)lists.linux.org.au > http://lists.linux.org.au/mailman/listinfo/lca-announce > >

1 0

OSD
by u15484＠ust-global.com 23 Jul '19

23 Jul '19

I have a three node cluster running on MIMIC one OSD has with ID 0 has taken out and data is cleared. Created OSD with same ID, however it is not coming up. See the status below. cephuser@CEP001:~/clusterstor$ sudo ceph osd status +----+-----------------+-------+-------+--------+---------+--------+---------+------------------+ | id | host | used | avail | wr ops | wr data | rd ops | rd data | state | +----+-----------------+-------+-------+--------+---------+--------+---------+------------------+ | 0 | | 0 | 0 | 0 | 0 | 0 | 0 | exists | | 1 | HostName | 1027M | 446G | 0 | 0 | 0 | 0 | exists,nodown,up | | 2 | HostName | 1039M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 3 | HostName | 1039M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 4 | HostName | 1039M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 5 | HostName | 1027M | 446G | 0 | 0 | 0 | 0 | exists,nodown,up | | 6 | HostName | 1039M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 7 | HostName | 1039M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 8 | HostName | 1039M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 9 | HostName | 1039M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 10 | HostName | 1028M | 446G | 0 | 0 | 0 | 0 | exists,nodown,up | | 11 | HostName | 1040M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 12 | HostName | 1040M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 13 | HostName | 1040M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | | 14 | HostName | 1040M | 3725G | 0 | 0 | 0 | 0 | exists,nodown,up | +----+-----------------+-------+-------+--------+---------+--------+---------+------------------+ Any help on this will be appreciated. Thanks

1 0

About osd_op_num_shards configuration
by 刘朝洋 22 Jul '19

22 Jul '19

Hello everybody! I test the performance of ceph cluster using different osd_op_num_shards and different osd_op_num_threads_per_shard configuration. And I found that using multi-thread(with a single shard) can get the same improvement of performance with multi-shard. However, the document of ceph osd config(http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/… said that using lower shard number may have deleterious effects. I want to know what the deleterious effects are. Thanks for the assistance! Fibird chaoyanglius(a)gmail.com

1 0

WAL on separate partition
by vladimir franciz blando 21 Jul '19

21 Jul '19

Hi, I have an existing Luminous installation, 3 nodes with 8x4TB HDD and a 1x200GB SSD which was used as journal before. On a default Luminous installation via ceph-deploy, I forgot to prepare the OSD with the WAL and DB on separate SSD. The environment is running and in production and I want to configure it to use the SSD as the WAL device or maybe for DB also, since the environment is in production I am hesitant to do it because it may cause some problems along the way. What should I do for me to reconfigure it without downtime, or if there are downtime at least minimal? ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 101.86957 root default -3 29.10559 host ceph01 0 hdd 3.63820 osd.0 up 1.00000 1.00000 1 hdd 3.63820 osd.1 up 1.00000 1.00000 2 hdd 3.63820 osd.2 up 1.00000 1.00000 3 hdd 3.63820 osd.3 up 1.00000 1.00000 4 hdd 3.63820 osd.4 up 1.00000 1.00000 5 hdd 3.63820 osd.5 up 1.00000 1.00000 6 hdd 3.63820 osd.6 up 1.00000 1.00000 7 hdd 3.63820 osd.7 up 1.00000 1.00000 -5 29.10559 host ceph02 8 hdd 3.63820 osd.8 up 1.00000 1.00000 9 hdd 3.63820 osd.9 up 1.00000 1.00000 10 hdd 3.63820 osd.10 up 1.00000 1.00000 11 hdd 3.63820 osd.11 up 1.00000 1.00000 12 hdd 3.63820 osd.12 up 1.00000 1.00000 13 hdd 3.63820 osd.13 up 1.00000 1.00000 14 hdd 3.63820 osd.14 up 1.00000 1.00000 15 hdd 3.63820 osd.15 up 1.00000 1.00000 -7 29.10559 host ceph03 16 hdd 3.63820 osd.16 up 1.00000 1.00000 17 hdd 3.63820 osd.17 up 1.00000 1.00000 18 hdd 3.63820 osd.18 up 1.00000 1.00000 19 hdd 3.63820 osd.19 up 1.00000 1.00000 20 hdd 3.63820 osd.20 up 1.00000 1.00000 21 hdd 3.63820 osd.21 up 1.00000 1.00000 22 hdd 3.63820 osd.22 up 1.00000 1.00000 23 hdd 3.63820 osd.23 up 1.00000 1.00000 - Vlad ᐧ

1 0

Help please
by Konstantin Pupkov 20 Jul '19

20 Jul '19

Hi! I am trying to setup ceph cluster from existing ubuntu boxes (I'll purchase it from VSP). How to create a cluster from it using existing folder like "/cephfs" in my Ubuntu box Thanks, Konstantin

1 0

Cephalopod layered image capabilities..
by ramanathan19591＠gmail.com 17 Jul '19

17 Jul '19

Hi team, I am pretty new into distributed storage world and learning ceph day by day. I run a openstack hypervisor with ceph as backend block device.. There is a need to backup the instances running inside openstack. I have explored an option of creating COW incremental snapshots on ceph for rbd objects and revert to it whenever something broke.. The snaps that are created in the ceph gets stored in ceph storage itself which makes it as a single point of failure. Now i need to know if there is any possibility to explicitly specify a location for the snapshot as to where it should be saved. Consider storing the snapshot alone on a separate server which has its own internal disks to eliminate single point of failure and reverting from it on the ceph storage cluster. Something like below to store only the layered image snap outside of ceph to a server to less consume the storage..?? $ rbd snap add (pool name)/(image name)@(snap name) user@10.x.x.x:/data I know that we can use the below command to store the current rbd image state and export it completely as independent image not as a layered snapshot which will take up exact space of the base rbd object. $ rbd -p (pool name) export (rbd image name)@(snap name) (local file name) Once its done and the image is stored outside can a snap revert be triggered from this whole image on ceph? While i agree that ceph as a distributed storage device is not designed basically to do a backup solutioning, i just need to know the possibilities of rbd image layering.. Any comments on my above questions is greatly appretiated.. Regards, Prasanna, Ceph-user.

1 1

cephfs client high latency rm, failed to xclock
by Andrej Filipčič 16 Jul '19

16 Jul '19

Hi, We are experiencing very slow "rm" on client lately, and it's not clear what is wrong. The symptom shows like this: on client: # cat /sys/kernel/debug/ceph/*/mdsc 3436737 mds0 rmdir #100216d0113/utils (hpc/session/H9MKDmgVn2unmmR0Xox1SiGmABFKDmABFKDmmH9XDmABFKDmzmjnGo/pilot/radical/utils) mds: # ceph daemon mds.velikaponca dump_ops_in_flight { "ops": [ { "description": "client_request(client.14321210:3436737 rmdir #0x100216d0113/utils 2019-07-03 09:48:02.000548 caller_uid=0, caller_gid=0{})", "initiated_at": "2019-07-03 09:48:02.001005", "age": 4.046726, "duration": 4.046759, "type_data": { "flag_point": "failed to xlock, waiting", "reqid": "client.14321210:3436737", "op_type": "client_request", "client_info": { "client": "client.14321210", "tid": 3436737 }, "events": [ { "time": "2019-07-03 09:48:02.001005", "event": "initiated" }, { "time": "2019-07-03 09:48:02.001005", "event": "header_read" }, { "time": "2019-07-03 09:48:02.001006", "event": "throttled" }, { "time": "2019-07-03 09:48:02.001011", "event": "all_read" }, { "time": "2019-07-03 09:48:02.001131", "event": "dispatched" }, { "time": "2019-07-03 09:48:02.001261", "event": "failed to wrlock, waiting" }, { "time": "2019-07-03 09:48:02.001744", "event": "failed to xlock, waiting" }, { "time": "2019-07-03 09:48:02.010963", "event": "failed to xlock, waiting" } ] } } ], "num_ops": 1 } The client requests appear to be very fast, but rmdir tends to fail to xclock frequently, and it takes up to 10s before the operation completes, which slows down a lot the services running on the client, and it's also not related to the load on ceph servers. Restarting mds solves the issue for few minutes, but then it reappears. The version is mimic 13.2.6, and the cluster is healthy, most of the clients are on 5.1.* kernel. The server experiencing the issues is on centos7 kernel 3.10.0-957.21.3.el7.x86_64, we have also tested it with 5.1.15 kernel showing the same symptoms. Any ideas how to solve this problem? Cheers, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 2

Multiple OSD crashes
by Daniel Aberger - Profihost AG 16 Jul '19

16 Jul '19

Hello, we are experiencing crashing OSDs in multiple independent Ceph clusters. Each OSD has very similar log entries regarding the crash as far as I can tell. Example log: https://pastebin.com/raw/vQ2AJ5ud I can provide you with more log files. They are too large for pastebin and I'm not aware of this mailing lists email attachement policy. Every log consists of the following entries: 2019-07-10 21:36:31.903886 7f322aeff700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = M key = 0x00000000000008c1'.0000461231.00000000000125574325' Value size = 184) Put( Prefix = M key = 0x00000000000008c1'._fastinfo' Value size = 186) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff6f002f0000'x' Value size = 325) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1608) Put( Prefix = L key = 0x000000000226dc7a Value size = 16440) 2019-07-10 21:36:31.913113 7f322aeff700 -1 /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f322aeff700 time 2019-07-10 21:36:31.903909 /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) ceph version 12.2.12-7-g1321c5e91f (1321c5e91f3d5d35dd5aa5a0029a54b9a8ab9498) luminous (stable) Unfortunately I'm unable to interpret the dumps. I hope you can help me with this issue. Regards, Daniel -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

1 0

Multiple OSD crashes
by Daniel Aberger - Profihost AG 16 Jul '19

16 Jul '19

Hello, we are experiencing crashing OSDs in multiple independent Ceph clusters. Each OSD has very similar log entries regarding the crash as far as I can tell. Example log: https://pastebin.com/raw/vQ2AJ5ud I can provide you with more log files. They are too large for pastebin and I'm not aware of this mailing lists email attachement policy. Every log consists of the following entries: 2019-07-10 21:36:31.903886 7f322aeff700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = M key = 0x00000000000008c1'.0000461231.00000000000125574325' Value size = 184) Put( Prefix = M key = 0x00000000000008c1'._fastinfo' Value size = 186) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff6f002f0000'x' Value size = 325) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1608) Put( Prefix = L key = 0x000000000226dc7a Value size = 16440) 2019-07-10 21:36:31.913113 7f322aeff700 -1 /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f322aeff700 time 2019-07-10 21:36:31.903909 /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) ceph version 12.2.12-7-g1321c5e91f (1321c5e91f3d5d35dd5aa5a0029a54b9a8ab9498) luminous (stable) Unfortunately I'm unable to interpret the dumps. I hope you can help me with this issue. Regards, Daniel -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

1 0

help
by masud parvez 11 Jul '19

11 Jul '19

I want to setup ceph install but facing this error when run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 0

ceph-deploy install error
by masud parvez 11 Jul '19

11 Jul '19

I want to setup ceph install but facing this error when run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw Now what can I do? <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 0

Is CephFS a good solution for the problem I have to solve?
by Joseph Pietras 11 Jul '19

11 Jul '19

As a brand newbie with CEPH I'm trying to figure out if it is a good solution given the hardware I have to work with and the problem I need solve. At first look CEPH does appear to do what is required, namely a replicating file system. But it appears it might be too "heavy". The environment is 2 Ubuntu Linux servers each with 2 hard drives. They will be clustered. Nothing, other than the hardware and OS, is decided. We can create any solution that works. One of the servers in the cluster will be primary and it will fail over to the secondary until the primary is brought back on line. Does Ceph make sense for keeping the hard drives/filesystems synced? Thanks for your input. Joseph The information transmitted is intended only for the person or entity to which it is addressed and may contain proprietary, business-confidential and/or privileged material. If you are not the intended recipient of this message you are hereby notified that any use, review, retransmission, dissemination, distribution, reproduction or any action taken in reliance upon this message is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 1

Ceph MON onde RAM recommandations
by Sebastian Wiesner 10 Jul '19

10 Jul '19

Hi community, we’re designing our first Ceph cluster here and we’re struggeling with the MON (and also MDS) nodes RAM recommendations. It sais „like 1GB per daemon instance“, but what daemons are meant here? The range is from per MON (MDS) daemon, which would be fairly low, to per OSD daemon, which would be fairly high. Any help would be appreciated. Kind regards Sebastian -- Sebastian Wiesner IT-Systemadministrator / Application Developer Department of Operative Processes and Systems (OPS) Dresden University of Technology Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden, Germany phone: +49 351 463-37828 fax: +49 351 463-32164 e-mail: sebastian.wiesner(a)tu-dresden.de WWW: http://www.tu-dresden.de/zih

3 3

Question on number of buckets per account
by Rajesh Vanga 05 Jul '19

05 Jul '19

Greetings, I was wondering if there are limitations or known performance concerns if we increase the number of buckets per account to more than 1000? Our usecase was to have close to 1500 buckets per account with approx 40k objects per bucket on average Thanks, Rajesh -- Save a tree. Don't print this e-mail unless it's really necessary.

1 0

Unable to increase pool pg_num
by Sylvain PORTIER 02 Jul '19

02 Jul '19

Hi all, I am using ceph 14.2.1 (Nautilus) I am unable to increase the pg_num of a pool. I have a pool named Backup, the current pg_num is 64 : ceph osd pool get Backup pg_num => result pg_num: 64 And when I try to increase it using the command ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512" And then I check with the command : ceph osd pool get Backup pg_num => result pg_num: 64 I don't how to increase the pg_num of a pool, I also tried the autoscale module, but it doesn't work (unable to activate the autoscale, always warn mode). Thank you for your help, Cabeur. --- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus

3 3

Ceph RBD High Latency
by mdunavant＠convokesystems.com 02 Jul '19

02 Jul '19

Hi all, I am having trouble with our cluster getting consistent RBD latencies to our KVM virtual machines connected via the KRBD driver. When measuring with tools like rbd perf image iotop, we constantly see latency spike up from around 1-2ms to 100+ms. This seems to kill our Windows VM SQL performance. I essentially have 2 questions: 1) Am I missing something with my configuration that should be applied to get consistent low latency to the VM guests? 2) When measuring the disks, it seems that sequential IO results in higher latency vs random IO. Is this correct or is there a way to tweak this using the KRBD driver? Configuration: 3 x MON/MGR nodes 12 x OSD nodes (24 x HDD, 2 x NVMe for DB and WAL) KVM clients attaching the RBD images via KRBD 1 pool w/ 16384 PGs Ceph version 14.2.1 Ceph.conf: [global] mon host = 10.97.11.17,10.97.11.27,10.97.11.37 public network = 10.97.11.0/24 cluster network = 10.97.12.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 30720 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 4096 osd pool default pgp num = 4096 osd crush chooseleaf type = 1 [osd] bluestore_default_buffered_write = false bluestore_default_buffered_read = true Thank you,

1 0

RBD High Latency
by Matt Dunavant 02 Jul '19

02 Jul '19

Hi all, I am having trouble with our cluster getting consistent RBD latencies to our KVM virtual machines connected via the KRBD driver. When measuring with tools like rbd perf image iotop, we constantly see latency spike up from around 1-2ms to 100+ms. This seems to kill our Windows VM SQL performance. I essentially have 2 questions: 1) Am I missing something with my configuration that should be applied to get consistent low latency to the VM guests? 2) When measuring the disks, it seems that sequential IO results in higher latency vs random IO. Is this correct or is there a way to tweak this using the KRBD driver? Configuration: 3 x MON/MGR nodes 12 x OSD nodes (24 x HDD, 2 x NVMe for DB and WAL) KVM clients attaching the RBD images via KRBD 1 pool w/ 16384 PGs Ceph version 14.2.1 Ceph.conf: [global] mon host = 10.97.11.17,10.97.11.27,10.97.11.37 public network = 10.97.11.0/24 cluster network = 10.97.12.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 30720 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 4096 osd pool default pgp num = 4096 osd crush chooseleaf type = 1 [osd] bluestore_default_buffered_write = false bluestore_default_buffered_read = true Thank you,

1 0

CephFS and RBD pool disk usage (objects inside)
by Bogdan Adrian Velica 01 Jul '19

01 Jul '19

Hello, Does anyone know a way I could get the actual disk usage of my objects inside RBD pool and CephFS pool? I am trying to understand if object X occupies 10G of space and object Y 20G of space and it's mounted by Alpha client or Beta client... Thank you very much for any input, Bogdan.

1 0

RBD Ceph terrible performance
by Jack Terry 01 Jul '19

01 Jul '19

I have been experimenting using Ceph mostly for KVM image storage but I just have been unable to get any decent RBD performance with sequential writes. Reads are mostly fine, same with random read/write they are handled fine. I appear to be limited to around 30-40MB/s write speed on 3 hard drives (size 2). I moved my tests over to a different single node and osd performance remained the same. I have messed with Bluestore cache, Rbd cache and the speed level remains fairly constant apart from a burst of speed when I begin the write test. Looking at iostat on the host, it appears the LVM groups for the osd-block are close to maxed out util (~ 80 % - 100 %) but the raw devices show anywhere between 10-40% and the write speed of a single drive should be a lot higher. During my various tests I have tried doing rados benchmarks and that gives me results way closer to what I was expecting. Also sometimes all the IO hits a single OSD, regardless of stripe size on RBD. After various trial and error I have tried putting the DB on an SSD and while it is not my ideal situtation (another point of failure), it appears to of helped, slightly. So one of my theories of it being the rockstore DB sync write was not the sole issue. However interestingly I noticed that CephFS gives me greater performance than RBD even just using a raw image file for a vm (random IO does suffer however), it is still not exactly what I would of hoped for my setup, around ~ 80MB-120MB on 3 osd - size 2. This was also the case with mimic and was one of the reasons I upgraded and remade all the bluestore OSDs, but without any significant gains. I was previously using ZFS and able to maintain over 200MB/s with a similar setup. I have been trying to crunch this issue for many weeks, however I feel I have just been going in circles and I wanted to use RBD as most of my data is VM storage. Using CephFS for the performance boost would mean I loose many of the advantages of RBD like snapshotting, ease of use between multiple pools, plus CephFS random write io is nowhere near as good. Is there any easy way to make RBD perform as well as CephFS, or to use cephfs with the above benefits. Ceph version ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)

1 0