- ceph-users - lists.ceph.io

WAL on separate partition
by vladimir franciz blando 21 Jul '19

21 Jul '19

Hi, I have an existing Luminous installation, 3 nodes with 8x4TB HDD and a 1x200GB SSD which was used as journal before. On a default Luminous installation via ceph-deploy, I forgot to prepare the OSD with the WAL and DB on separate SSD. The environment is running and in production and I want to configure it to use the SSD as the WAL device or maybe for DB also, since the environment is in production I am hesitant to do it because it may cause some problems along the way. What should I do for me to reconfigure it without downtime, or if there are downtime at least minimal? ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 101.86957 root default -3 29.10559 host ceph01 0 hdd 3.63820 osd.0 up 1.00000 1.00000 1 hdd 3.63820 osd.1 up 1.00000 1.00000 2 hdd 3.63820 osd.2 up 1.00000 1.00000 3 hdd 3.63820 osd.3 up 1.00000 1.00000 4 hdd 3.63820 osd.4 up 1.00000 1.00000 5 hdd 3.63820 osd.5 up 1.00000 1.00000 6 hdd 3.63820 osd.6 up 1.00000 1.00000 7 hdd 3.63820 osd.7 up 1.00000 1.00000 -5 29.10559 host ceph02 8 hdd 3.63820 osd.8 up 1.00000 1.00000 9 hdd 3.63820 osd.9 up 1.00000 1.00000 10 hdd 3.63820 osd.10 up 1.00000 1.00000 11 hdd 3.63820 osd.11 up 1.00000 1.00000 12 hdd 3.63820 osd.12 up 1.00000 1.00000 13 hdd 3.63820 osd.13 up 1.00000 1.00000 14 hdd 3.63820 osd.14 up 1.00000 1.00000 15 hdd 3.63820 osd.15 up 1.00000 1.00000 -7 29.10559 host ceph03 16 hdd 3.63820 osd.16 up 1.00000 1.00000 17 hdd 3.63820 osd.17 up 1.00000 1.00000 18 hdd 3.63820 osd.18 up 1.00000 1.00000 19 hdd 3.63820 osd.19 up 1.00000 1.00000 20 hdd 3.63820 osd.20 up 1.00000 1.00000 21 hdd 3.63820 osd.21 up 1.00000 1.00000 22 hdd 3.63820 osd.22 up 1.00000 1.00000 23 hdd 3.63820 osd.23 up 1.00000 1.00000 - Vlad ᐧ

1 0

Help please
by Konstantin Pupkov 19 Jul '19

19 Jul '19

Hi! I am trying to setup ceph cluster from existing ubuntu boxes (I'll purchase it from VSP). How to create a cluster from it using existing folder like "/cephfs" in my Ubuntu box Thanks, Konstantin

1 0

Cephalopod layered image capabilities..
by ramanathan19591＠gmail.com 17 Jul '19

17 Jul '19

Hi team, I am pretty new into distributed storage world and learning ceph day by day. I run a openstack hypervisor with ceph as backend block device.. There is a need to backup the instances running inside openstack. I have explored an option of creating COW incremental snapshots on ceph for rbd objects and revert to it whenever something broke.. The snaps that are created in the ceph gets stored in ceph storage itself which makes it as a single point of failure. Now i need to know if there is any possibility to explicitly specify a location for the snapshot as to where it should be saved. Consider storing the snapshot alone on a separate server which has its own internal disks to eliminate single point of failure and reverting from it on the ceph storage cluster. Something like below to store only the layered image snap outside of ceph to a server to less consume the storage..?? $ rbd snap add (pool name)/(image name)@(snap name) user@10.x.x.x:/data I know that we can use the below command to store the current rbd image state and export it completely as independent image not as a layered snapshot which will take up exact space of the base rbd object. $ rbd -p (pool name) export (rbd image name)@(snap name) (local file name) Once its done and the image is stored outside can a snap revert be triggered from this whole image on ceph? While i agree that ceph as a distributed storage device is not designed basically to do a backup solutioning, i just need to know the possibilities of rbd image layering.. Any comments on my above questions is greatly appretiated.. Regards, Prasanna, Ceph-user.

1 1

cephfs client high latency rm, failed to xclock
by Andrej Filipčič 16 Jul '19

16 Jul '19

Hi, We are experiencing very slow "rm" on client lately, and it's not clear what is wrong. The symptom shows like this: on client: # cat /sys/kernel/debug/ceph/*/mdsc 3436737 mds0 rmdir #100216d0113/utils (hpc/session/H9MKDmgVn2unmmR0Xox1SiGmABFKDmABFKDmmH9XDmABFKDmzmjnGo/pilot/radical/utils) mds: # ceph daemon mds.velikaponca dump_ops_in_flight { "ops": [ { "description": "client_request(client.14321210:3436737 rmdir #0x100216d0113/utils 2019-07-03 09:48:02.000548 caller_uid=0, caller_gid=0{})", "initiated_at": "2019-07-03 09:48:02.001005", "age": 4.046726, "duration": 4.046759, "type_data": { "flag_point": "failed to xlock, waiting", "reqid": "client.14321210:3436737", "op_type": "client_request", "client_info": { "client": "client.14321210", "tid": 3436737 }, "events": [ { "time": "2019-07-03 09:48:02.001005", "event": "initiated" }, { "time": "2019-07-03 09:48:02.001005", "event": "header_read" }, { "time": "2019-07-03 09:48:02.001006", "event": "throttled" }, { "time": "2019-07-03 09:48:02.001011", "event": "all_read" }, { "time": "2019-07-03 09:48:02.001131", "event": "dispatched" }, { "time": "2019-07-03 09:48:02.001261", "event": "failed to wrlock, waiting" }, { "time": "2019-07-03 09:48:02.001744", "event": "failed to xlock, waiting" }, { "time": "2019-07-03 09:48:02.010963", "event": "failed to xlock, waiting" } ] } } ], "num_ops": 1 } The client requests appear to be very fast, but rmdir tends to fail to xclock frequently, and it takes up to 10s before the operation completes, which slows down a lot the services running on the client, and it's also not related to the load on ceph servers. Restarting mds solves the issue for few minutes, but then it reappears. The version is mimic 13.2.6, and the cluster is healthy, most of the clients are on 5.1.* kernel. The server experiencing the issues is on centos7 kernel 3.10.0-957.21.3.el7.x86_64, we have also tested it with 5.1.15 kernel showing the same symptoms. Any ideas how to solve this problem? Cheers, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 2

Multiple OSD crashes
by Daniel Aberger - Profihost AG 16 Jul '19

16 Jul '19

Hello, we are experiencing crashing OSDs in multiple independent Ceph clusters. Each OSD has very similar log entries regarding the crash as far as I can tell. Example log: https://pastebin.com/raw/vQ2AJ5ud I can provide you with more log files. They are too large for pastebin and I'm not aware of this mailing lists email attachement policy. Every log consists of the following entries: 2019-07-10 21:36:31.903886 7f322aeff700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = M key = 0x00000000000008c1'.0000461231.00000000000125574325' Value size = 184) Put( Prefix = M key = 0x00000000000008c1'._fastinfo' Value size = 186) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff6f002f0000'x' Value size = 325) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1608) Put( Prefix = L key = 0x000000000226dc7a Value size = 16440) 2019-07-10 21:36:31.913113 7f322aeff700 -1 /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f322aeff700 time 2019-07-10 21:36:31.903909 /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) ceph version 12.2.12-7-g1321c5e91f (1321c5e91f3d5d35dd5aa5a0029a54b9a8ab9498) luminous (stable) Unfortunately I'm unable to interpret the dumps. I hope you can help me with this issue. Regards, Daniel -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

1 0

Multiple OSD crashes
by Daniel Aberger - Profihost AG 16 Jul '19

16 Jul '19

Hello, we are experiencing crashing OSDs in multiple independent Ceph clusters. Each OSD has very similar log entries regarding the crash as far as I can tell. Example log: https://pastebin.com/raw/vQ2AJ5ud I can provide you with more log files. They are too large for pastebin and I'm not aware of this mailing lists email attachement policy. Every log consists of the following entries: 2019-07-10 21:36:31.903886 7f322aeff700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = M key = 0x00000000000008c1'.0000461231.00000000000125574325' Value size = 184) Put( Prefix = M key = 0x00000000000008c1'._fastinfo' Value size = 186) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff6f002f0000'x' Value size = 325) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1608) Put( Prefix = L key = 0x000000000226dc7a Value size = 16440) 2019-07-10 21:36:31.913113 7f322aeff700 -1 /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f322aeff700 time 2019-07-10 21:36:31.903909 /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) ceph version 12.2.12-7-g1321c5e91f (1321c5e91f3d5d35dd5aa5a0029a54b9a8ab9498) luminous (stable) Unfortunately I'm unable to interpret the dumps. I hope you can help me with this issue. Regards, Daniel -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

1 0

help
by masud parvez 11 Jul '19

11 Jul '19

I want to setup ceph install but facing this error when run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 0

ceph-deploy install error
by masud parvez 11 Jul '19

11 Jul '19

I want to setup ceph install but facing this error when run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw Now what can I do? <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 0

Is CephFS a good solution for the problem I have to solve?
by Joseph Pietras 11 Jul '19

11 Jul '19

As a brand newbie with CEPH I'm trying to figure out if it is a good solution given the hardware I have to work with and the problem I need solve. At first look CEPH does appear to do what is required, namely a replicating file system. But it appears it might be too "heavy". The environment is 2 Ubuntu Linux servers each with 2 hard drives. They will be clustered. Nothing, other than the hardware and OS, is decided. We can create any solution that works. One of the servers in the cluster will be primary and it will fail over to the secondary until the primary is brought back on line. Does Ceph make sense for keeping the hard drives/filesystems synced? Thanks for your input. Joseph The information transmitted is intended only for the person or entity to which it is addressed and may contain proprietary, business-confidential and/or privileged material. If you are not the intended recipient of this message you are hereby notified that any use, review, retransmission, dissemination, distribution, reproduction or any action taken in reliance upon this message is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 1

Ceph MON onde RAM recommandations
by Sebastian Wiesner 10 Jul '19

10 Jul '19

Hi community, we’re designing our first Ceph cluster here and we’re struggeling with the MON (and also MDS) nodes RAM recommendations. It sais „like 1GB per daemon instance“, but what daemons are meant here? The range is from per MON (MDS) daemon, which would be fairly low, to per OSD daemon, which would be fairly high. Any help would be appreciated. Kind regards Sebastian -- Sebastian Wiesner IT-Systemadministrator / Application Developer Department of Operative Processes and Systems (OPS) Dresden University of Technology Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden, Germany phone: +49 351 463-37828 fax: +49 351 463-32164 e-mail: sebastian.wiesner(a)tu-dresden.de WWW: http://www.tu-dresden.de/zih

3 3