- ceph-users - lists.ceph.io

Cephalopod layered image capabilities..
by ramanathan19591＠gmail.com 17 Jul '19

17 Jul '19

Hi team, I am pretty new into distributed storage world and learning ceph day by day. I run a openstack hypervisor with ceph as backend block device.. There is a need to backup the instances running inside openstack. I have explored an option of creating COW incremental snapshots on ceph for rbd objects and revert to it whenever something broke.. The snaps that are created in the ceph gets stored in ceph storage itself which makes it as a single point of failure. Now i need to know if there is any possibility to explicitly specify a location for the snapshot as to where it should be saved. Consider storing the snapshot alone on a separate server which has its own internal disks to eliminate single point of failure and reverting from it on the ceph storage cluster. Something like below to store only the layered image snap outside of ceph to a server to less consume the storage..?? $ rbd snap add (pool name)/(image name)@(snap name) user@10.x.x.x:/data I know that we can use the below command to store the current rbd image state and export it completely as independent image not as a layered snapshot which will take up exact space of the base rbd object. $ rbd -p (pool name) export (rbd image name)@(snap name) (local file name) Once its done and the image is stored outside can a snap revert be triggered from this whole image on ceph? While i agree that ceph as a distributed storage device is not designed basically to do a backup solutioning, i just need to know the possibilities of rbd image layering.. Any comments on my above questions is greatly appretiated.. Regards, Prasanna, Ceph-user.

1 1

cephfs client high latency rm, failed to xclock
by Andrej Filipčič 16 Jul '19

16 Jul '19

Hi, We are experiencing very slow "rm" on client lately, and it's not clear what is wrong. The symptom shows like this: on client: # cat /sys/kernel/debug/ceph/*/mdsc 3436737 mds0 rmdir #100216d0113/utils (hpc/session/H9MKDmgVn2unmmR0Xox1SiGmABFKDmABFKDmmH9XDmABFKDmzmjnGo/pilot/radical/utils) mds: # ceph daemon mds.velikaponca dump_ops_in_flight { "ops": [ { "description": "client_request(client.14321210:3436737 rmdir #0x100216d0113/utils 2019-07-03 09:48:02.000548 caller_uid=0, caller_gid=0{})", "initiated_at": "2019-07-03 09:48:02.001005", "age": 4.046726, "duration": 4.046759, "type_data": { "flag_point": "failed to xlock, waiting", "reqid": "client.14321210:3436737", "op_type": "client_request", "client_info": { "client": "client.14321210", "tid": 3436737 }, "events": [ { "time": "2019-07-03 09:48:02.001005", "event": "initiated" }, { "time": "2019-07-03 09:48:02.001005", "event": "header_read" }, { "time": "2019-07-03 09:48:02.001006", "event": "throttled" }, { "time": "2019-07-03 09:48:02.001011", "event": "all_read" }, { "time": "2019-07-03 09:48:02.001131", "event": "dispatched" }, { "time": "2019-07-03 09:48:02.001261", "event": "failed to wrlock, waiting" }, { "time": "2019-07-03 09:48:02.001744", "event": "failed to xlock, waiting" }, { "time": "2019-07-03 09:48:02.010963", "event": "failed to xlock, waiting" } ] } } ], "num_ops": 1 } The client requests appear to be very fast, but rmdir tends to fail to xclock frequently, and it takes up to 10s before the operation completes, which slows down a lot the services running on the client, and it's also not related to the load on ceph servers. Restarting mds solves the issue for few minutes, but then it reappears. The version is mimic 13.2.6, and the cluster is healthy, most of the clients are on 5.1.* kernel. The server experiencing the issues is on centos7 kernel 3.10.0-957.21.3.el7.x86_64, we have also tested it with 5.1.15 kernel showing the same symptoms. Any ideas how to solve this problem? Cheers, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

1 2

Multiple OSD crashes
by Daniel Aberger - Profihost AG 16 Jul '19

16 Jul '19

Hello, we are experiencing crashing OSDs in multiple independent Ceph clusters. Each OSD has very similar log entries regarding the crash as far as I can tell. Example log: https://pastebin.com/raw/vQ2AJ5ud I can provide you with more log files. They are too large for pastebin and I'm not aware of this mailing lists email attachement policy. Every log consists of the following entries: 2019-07-10 21:36:31.903886 7f322aeff700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = M key = 0x00000000000008c1'.0000461231.00000000000125574325' Value size = 184) Put( Prefix = M key = 0x00000000000008c1'._fastinfo' Value size = 186) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff6f002f0000'x' Value size = 325) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1608) Put( Prefix = L key = 0x000000000226dc7a Value size = 16440) 2019-07-10 21:36:31.913113 7f322aeff700 -1 /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f322aeff700 time 2019-07-10 21:36:31.903909 /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) ceph version 12.2.12-7-g1321c5e91f (1321c5e91f3d5d35dd5aa5a0029a54b9a8ab9498) luminous (stable) Unfortunately I'm unable to interpret the dumps. I hope you can help me with this issue. Regards, Daniel -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

1 0

Multiple OSD crashes
by Daniel Aberger - Profihost AG 16 Jul '19

16 Jul '19

Hello, we are experiencing crashing OSDs in multiple independent Ceph clusters. Each OSD has very similar log entries regarding the crash as far as I can tell. Example log: https://pastebin.com/raw/vQ2AJ5ud I can provide you with more log files. They are too large for pastebin and I'm not aware of this mailing lists email attachement policy. Every log consists of the following entries: 2019-07-10 21:36:31.903886 7f322aeff700 -1 rocksdb: submit_transaction error: Corruption: block checksum mismatch code = 2 Rocksdb transaction: Put( Prefix = M key = 0x00000000000008c1'.0000461231.00000000000125574325' Value size = 184) Put( Prefix = M key = 0x00000000000008c1'._fastinfo' Value size = 186) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff6f002f0000'x' Value size = 325) Put( Prefix = O key = 0x7f80000000000000015806b4'(!rbd_data.7c012a6b8b4567.000000000000004e!='0xfffffffffffffffeffffffffffffffff'o' Value size = 1608) Put( Prefix = L key = 0x000000000226dc7a Value size = 16440) 2019-07-10 21:36:31.913113 7f322aeff700 -1 /build/ceph/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f322aeff700 time 2019-07-10 21:36:31.903909 /build/ceph/src/os/bluestore/BlueStore.cc: 8808: FAILED assert(r == 0) ceph version 12.2.12-7-g1321c5e91f (1321c5e91f3d5d35dd5aa5a0029a54b9a8ab9498) luminous (stable) Unfortunately I'm unable to interpret the dumps. I hope you can help me with this issue. Regards, Daniel -- Mit freundlichen Grüßen Daniel Aberger Ihr Profihost Team ------------------------------- Profihost AG Expo Plaza 1 30539 Hannover Deutschland Tel.: +49 (511) 5151 8181 | Fax.: +49 (511) 5151 8282 URL: http://www.profihost.com | E-Mail: info(a)profihost.com Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827 Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350 Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)

1 0

help
by masud parvez 11 Jul '19

11 Jul '19

I want to setup ceph install but facing this error when run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 0

ceph-deploy install error
by masud parvez 11 Jul '19

11 Jul '19

I want to setup ceph install but facing this error when run this command #ceph-deploy install ceph-deploy monnode1 osd0 osd1 [ceph-deploy][WARNIN] E: Sub-process /usr/bin/dpkg returned an error code (1) [ceph-deploy][ERROR ] RuntimeError: command returned non-zero exit status: 100 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install -o Dpkg::Options::=--force-confnew ceph ceph-osd ceph-mds ceph-mon radosgw Now what can I do? <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 0

Is CephFS a good solution for the problem I have to solve?
by Joseph Pietras 11 Jul '19

11 Jul '19

As a brand newbie with CEPH I'm trying to figure out if it is a good solution given the hardware I have to work with and the problem I need solve. At first look CEPH does appear to do what is required, namely a replicating file system. But it appears it might be too "heavy". The environment is 2 Ubuntu Linux servers each with 2 hard drives. They will be clustered. Nothing, other than the hardware and OS, is decided. We can create any solution that works. One of the servers in the cluster will be primary and it will fail over to the secondary until the primary is brought back on line. Does Ceph make sense for keeping the hard drives/filesystems synced? Thanks for your input. Joseph The information transmitted is intended only for the person or entity to which it is addressed and may contain proprietary, business-confidential and/or privileged material. If you are not the intended recipient of this message you are hereby notified that any use, review, retransmission, dissemination, distribution, reproduction or any action taken in reliance upon this message is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 1

Ceph MON onde RAM recommandations
by Sebastian Wiesner 10 Jul '19

10 Jul '19

Hi community, we’re designing our first Ceph cluster here and we’re struggeling with the MON (and also MDS) nodes RAM recommendations. It sais „like 1GB per daemon instance“, but what daemons are meant here? The range is from per MON (MDS) daemon, which would be fairly low, to per OSD daemon, which would be fairly high. Any help would be appreciated. Kind regards Sebastian -- Sebastian Wiesner IT-Systemadministrator / Application Developer Department of Operative Processes and Systems (OPS) Dresden University of Technology Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden, Germany phone: +49 351 463-37828 fax: +49 351 463-32164 e-mail: sebastian.wiesner(a)tu-dresden.de WWW: http://www.tu-dresden.de/zih

3 3

Question on number of buckets per account
by Rajesh Vanga 05 Jul '19

05 Jul '19

Greetings, I was wondering if there are limitations or known performance concerns if we increase the number of buckets per account to more than 1000? Our usecase was to have close to 1500 buckets per account with approx 40k objects per bucket on average Thanks, Rajesh -- Save a tree. Don't print this e-mail unless it's really necessary.

1 0

Unable to increase pool pg_num
by Sylvain PORTIER 02 Jul '19

02 Jul '19

Hi all, I am using ceph 14.2.1 (Nautilus) I am unable to increase the pg_num of a pool. I have a pool named Backup, the current pg_num is 64 : ceph osd pool get Backup pg_num => result pg_num: 64 And when I try to increase it using the command ceph osd pool set Backup pg_num 512 => result "set pool 6 pg_num to 512" And then I check with the command : ceph osd pool get Backup pg_num => result pg_num: 64 I don't how to increase the pg_num of a pool, I also tried the autoscale module, but it doesn't work (unable to activate the autoscale, always warn mode). Thank you for your help, Cabeur. --- L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast. https://www.avast.com/antivirus

3 3