- ceph-users - lists.ceph.io

cephadm - How to deploy ceph cluster with a partition on SSD for block.db
by klemen＠psi-net.si 11 Sep '20

11 Sep '20

I'm trying to deploy a ceph cluster with a cephadm tool. I've already successfully done all steps except adding OSDs. My testing equipment consists of three hosts. Each host has SSD storage, where OS is installed into. On that storage I created partition, which can be used as a ceph block.db. Hosts have also 2 additional HDs (spinning drives) for OSD data. On docs I couldn't find how to deploy such configuration. Do you have any hints, how to do that? Thanks for help!

5 7

slow "rados ls"
by Marcel Kuiper 11 Sep '20

11 Sep '20

Hi One of my clusters running nautilus 14.2.8 is very slow (13 seconds or so where my other clusters are returning almost instantanious) when doing a 'rados --pool rc3-se.rgw.buckets.index ls' from one of the monitors. I checked - ceph status => OK - routing to/from osds ok (I see a lot of established connections to osds due to the command, nothing in syn_sent indicating incomplete handshake) - ping times are OK - no interface errors - no packet drops - no increasing send queus - and as far as I can see nothing out of the ordinary in mon and osd logs I have no clue how to debug the issue. If someone has pointers it would be much appreciated Kind Regards Marcel

4 16

Re: ceph-osd performance on ram disk
by George Shuklin 11 Sep '20

11 Sep '20

Thank you! I know that article, but they promise 6 core use per OSD, and I got barely over three, and all this in totally synthetic environment with no SDD to blame (brd is more than fast and have a very consistent latency under any kind of load). On Thu, Sep 10, 2020, 19:39 Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > Hi George, > > Very interesting and also a bit expecting result. Some messages posted > here are already indicating that getting expensive top of the line > hardware does not really result in any performance increase above some > level. Vitaliy has documented something similar[1] > > [1] > https://yourcmc.ru/wiki/Ceph_performance > > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: [ceph-users] ceph-osd performance on ram disk > > I'm creating a benchmark suite for Сeph. > > During benchmarking of benchmark, I've checked how fast ceph-osd works. > I decided to skip all 'SSD mess' and use brd (block ram disk, modprobe > brd) as underlying storage. Brd itself can yield up to 2.7Mpps in fio. > In single thread mode (iodepth=1) it can yield up to 750k IOPS. LVM over > brd gives about 600kIOPS in single-threaded mode with iodepth=1 (16us > latency). > > But, as soon as I put ceph-osd (bluestore) on it, I see something very > odd. No matter how much parallel load I push onto this OSD, it never > gives more than 30 kIOPS, and I can't understand where bottleneck is. > > CPU utilization: ~300%. There are 8 cores on my setup, so, CPU is not a > bottleneck. > > Network: I've moved benchmark on the same host as OSD, so it's a > localhost. Even counting network, it's still far away from saturation. > 30kIOPS (4k) is about 1Gb/s, but I have 10G links. Anyway, tests are run > on localhost, so network is irrelevant (I've checked it, traffic is on > localhost). Test itself consumes about 70% CPU of one core, so there are > plenty left. > > Replication: I've killed it (size=1, single osd in the pool). > > single-threaded latency: 200us, 4.8kIOPS. > iopdeth=32: 2ms (15kIOPS). > iodepth=16,numjobs=8: 5ms (24k IOPS) > > I'm running fio with 'rados' ioengine, and it looks like putting more > workers doesn't change much, so it's not rados ioengine. > > As there is plenty CPU and IO left, there is only one possible place for > bottleneck: some time-consuming single-threaded code in ceph-osd. > > Are there any knobs to tweak to see higher performance for ceph-osd? I'm > pretty sure it's not any kind of leveling, GC or other 'iops-related' > issues (brd has performance of two order of magnitude higher). > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >

3 6

Printer is in error state because of motherboard malfunction? Contact customer care.
by mary smith 11 Sep '20

11 Sep '20

Sometimes the motherboard glitch can cause printer is in error state situation. Therefore, you can use the troubleshooting solutions that are available in the tech consultancy websites or you can call the help team and use their assistance to deal with the issue. In addition to that, you can always watch some tech trick videos on Youtube. https://www.epsonprintersupportpro.net/epson-printer-in-error-state/

1 0

Errror in Facebook drafts? Find support by dialing Facebook Customer Service Toll Free Number.
by mary smith 11 Sep '20

11 Sep '20

The drafts are the place the message gets set aside in the occasion that you've excluded the recipient. In any case, if there's a glitch there, by then you can get it cured by using the help from the tech help destinations or you can dial the Facebook Customer Service Toll Free Number and have a conversation with the rep of the help gathering. https://www.customercare-email.com/facebook-customer-service.html

1 0

ceph config dump question
by Dave Baukus 10 Sep '20

10 Sep '20

A naive ceph user asks: I have 3 node cluster configured with 72 bluestore OSDs running on Ubuntu 20.01, Ceph Octopus 15.2.4 The cluster is configured via ceph-ansible stable-5.0. No configuration changes have been made outside of what is generated by ceph-ansible. I expected "ceph config dump" to show the entire configuration: # ceph config dump WHO MASK LEVEL OPTION VALUE RO # echo $? 0 Is my expectation wrong or is something broken ? If I set a value via ceph then it is reflected in the "ceph config dump" output # ceph config set osd osd_memory_target 2147483648 # ceph config dump WHO MASK LEVEL OPTION VALUE RO osd basic osd_memory_target 2147483648 If my expectation is wrong then how does one view the configuration defaults and all ? Thanks, -- Dave Baukus

1 0

Multipart uploads with partsizes larger than 16MiB failing on Nautilus
by shubjero 10 Sep '20

10 Sep '20

Hey all, I'm creating a new post for this issue as we've narrowed the problem down to a partsize limitation on multipart upload. We have discovered that in our production Nautilus (14.2.11) cluster and our lab Nautilus (14.2.10) cluster that multipart uploads with a configured part size of greater than 16777216 bytes (16MiB) will return a status 500 / internal server error from radosgw. So far I have increased the following rgw settings/values that looked suspect, without any success/improvement with partsizes. Such as: "rgw_get_obj_window_size": "16777216", "rgw_put_obj_min_window_size": "16777216", I am trying to determine if this is because of a conservative default setting somewhere that I don't know about or if this is perhaps a bug? I would appreciate it if someone on Nautilus with rgw could also test / provide feedback. It's very easy to reproduce and configuring your partsize with aws2cli requires you to put the following in your aws 'config' s3 = multipart_chunksize = 32MB rgw server logs during a failed multipart upload (32MB chunk/partsize): 2020-09-08 15:59:36.054 7f2d32fa6700 1 ====== starting new request req=0x55953dc36930 ===== 2020-09-08 15:59:36.082 7f2d32fa6700 -1 res_query() failed 2020-09-08 15:59:36.138 7f2d32fa6700 1 ====== req done req=0x55953dc36930 op status=0 http_status=200 latency=0.0839988s ====== 2020-09-08 16:00:07.285 7f2d3dfbc700 1 ====== starting new request req=0x55953dc36930 ===== 2020-09-08 16:00:07.285 7f2d3dfbc700 -1 res_query() failed 2020-09-08 16:00:07.353 7f2d00741700 1 ====== starting new request req=0x55954dd5e930 ===== 2020-09-08 16:00:07.357 7f2d00741700 -1 res_query() failed 2020-09-08 16:00:07.413 7f2cc56cb700 1 ====== starting new request req=0x55953dc02930 ===== 2020-09-08 16:00:07.417 7f2cc56cb700 -1 res_query() failed 2020-09-08 16:00:07.473 7f2cb26a5700 1 ====== starting new request req=0x5595426f6930 ===== 2020-09-08 16:00:07.473 7f2cb26a5700 -1 res_query() failed 2020-09-08 16:00:09.465 7f2d3dfbc700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.465 7f2d3dfbc700 1 ====== req done req=0x55953dc36930 op status=-35 http_status=500 latency=2.17997s ====== 2020-09-08 16:00:09.549 7f2d00741700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.549 7f2d00741700 1 ====== req done req=0x55954dd5e930 op status=-35 http_status=500 latency=2.19597s ====== 2020-09-08 16:00:09.605 7f2cc56cb700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.609 7f2cc56cb700 1 ====== req done req=0x55953dc02930 op status=-35 http_status=500 latency=2.19597s ====== 2020-09-08 16:00:09.641 7f2cb26a5700 0 WARNING: set_req_state_err err_no=35 resorting to 500 2020-09-08 16:00:09.641 7f2cb26a5700 1 ====== req done req=0x5595426f6930 op status=-35 http_status=500 latency=2.16797s ====== awscli client side output during a failed multipart upload: root@jump:~# aws --no-verify-ssl --endpoint-url http://lab-object.cancercollaboratory.org:7480 s3 cp 4GBfile s3://troubleshooting upload failed: ./4GBfile to s3://troubleshooting/4GBfile An error occurred (UnknownError) when calling the UploadPart operation (reached max retries: 2): Unknown Thanks, Jared Baker Cloud Architect for the Cancer Genome Collaboratory Ontario Institute for Cancer Research

3 6

Octopus dashboard: rbd-mirror page shows error for primary site
by Eugen Block 10 Sep '20

10 Sep '20

Hi *, I was just testing rbd-mirror on ceph version 15.2.4-864-g0f510cb110 (0f510cb1101879a5941dfa1fa824bf97db6c3d08) octopus (stable) and noticed mgr errors on the primary site (also in version 15.2.2): ---snip--- 2020-09-10T11:20:01.724+0200 7f1c1b46a700 0 [dashboard ERROR controllers.rbd_mirror] Failed to list mirror image status rbd-pool1 Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in _get_pool_datum for image in mirror_image_status File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in <listcomp> for image in mirror_image_status KeyError: 'description' 2020-09-10T11:20:01.724+0200 7f1c1b46a700 0 [dashboard ERROR viewcache] Error while calling fn=<function _get_pool_datum at 0x7f1c5ac58f28> ex='description' Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/tools.py", line 156, in run val = self.fn(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in _get_pool_datum for image in mirror_image_status File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in <listcomp> for image in mirror_image_status KeyError: 'description' 2020-09-10T11:20:01.724+0200 7f1c1ec71700 0 [dashboard ERROR viewcache] Error while calling fn=<function _get_content_data at 0x7f1c5ac610d0> ex='description' Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/tools.py", line 156, in run val = self.fn(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 283, in _get_content_data _, pool = _get_pool_datum(pool_name) File "/usr/share/ceph/mgr/dashboard/tools.py", line 254, in wrapper return rvc.run(fn, args, kwargs) File "/usr/share/ceph/mgr/dashboard/tools.py", line 236, in run raise self.exception File "/usr/share/ceph/mgr/dashboard/tools.py", line 156, in run val = self.fn(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in _get_pool_datum for image in mirror_image_status File "/usr/share/ceph/mgr/dashboard/controllers/rbd_mirroring.py", line 259, in <listcomp> for image in mirror_image_status KeyError: 'description' 2020-09-10T11:20:01.724+0200 7f1c23ebb700 0 [dashboard ERROR request] [X.X.X.X:42050] [GET] [500] [0.020s] [admin] [513.0B] /api/block/mirroring/summary 2020-09-10T11:20:01.728+0200 7f1c23ebb700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "64ea5407-171b-4f6c-862e-fc881e68fec0"} ---snip--- The mirroring itself seems to work fine, I haven't noticed issues with the image on the secondary site. I also uploaded a screenshot of the dashboard [1], the installation is on openSUSE Leap 15.2. I couldn't find a tracker issue or a report in bugzilla.opensuse.org, but before creating a ticket I wanted to ask if this is a known issue. The rbd-mirror panel works fine on the secondary site, I assume because there is actually an rbd-mirror daemon running. Or maybe I just misunderstand the purpose, but if there is no mirror daemon on site A there shouldn't be an error message, should it? Regards, Eugen [1] https://paste.opensuse.org/36442780

2 3

Octopus: snapshot errors during rbd import
by Eugen Block 10 Sep '20

10 Sep '20

Hi *, I'm currently testing rbd-mirror on ceph version 15.2.4-864-g0f510cb110 (0f510cb1101879a5941dfa1fa824bf97db6c3d08) octopus (stable) and saw this during an rbd import of a fresh image on the primary site: ---snip--- ceph1:~ # rbd import /mnt/SUSE-OPENSTACK-CLOUD-7-x86_64-GM-DVD1.iso rbd-pool1/cloud7 Importing image: 5% complete...2020-09-10T12:00:10.220+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c022060 send_op: snapshot doesn't exist 2020-09-10T12:00:10.224+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c022060 should_complete: encountered error: (2) No such file or directory 2020-09-10T12:00:10.232+0200 7ff1ba7fc700 -1 librbd::Operations: No such snapshot found. Importing image: 6% complete...2020-09-10T12:00:13.940+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c032eb0 send_op: snapshot doesn't exist 2020-09-10T12:00:13.940+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c032eb0 should_complete: encountered error: (2) No such file or directory Importing image: 7% complete...2020-09-10T12:00:18.596+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c027af0 send_op: snapshot doesn't exist 2020-09-10T12:00:18.596+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c027af0 should_complete: encountered error: (2) No such file or directory Importing image: 8% complete...2020-09-10T12:00:24.048+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c02f940 send_op: snapshot doesn't exist 2020-09-10T12:00:24.048+0200 7ff1b9ffb700 -1 librbd::SnapshotRemoveRequest: 0x7ff19c02f940 should_complete: encountered error: (2) No such file or directory Importing image: 100% complete...done. ---snip--- The import eventually succeeds and the image is fine (md5sum on the remote image is correct), but I'm wondering what snapshot it is looking for since there aren't any (yet). Here's some info about the image (not sure if it's relevant): ---snip--- ceph1:~ # rbd info rbd-pool1/cloud7 rbd image 'cloud7': size 1.4 GiB in 347 objects order 22 (4 MiB objects) snapshot_count: 0 id: 149e4eb8e387a block_name_prefix: rbd_data.149e4eb8e387a format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling op_features: flags: create_timestamp: Thu Sep 10 11:59:18 2020 access_timestamp: Thu Sep 10 11:59:18 2020 modify_timestamp: Thu Sep 10 12:03:30 2020 journal: 149e4eb8e387a mirroring state: enabled mirroring mode: journal mirroring global id: 82bb30ec-bd33-4f84-9d0c-b3650b567670 mirroring primary: true ---snip--- Is this worth a bug report or is it already known and I just couldn't find anything? Regards, Eugen

2 2

Re: Moving OSD from one node to another
by Eugen Block 10 Sep '20

10 Sep '20

Hi, I haven't done this myself yet but you should be able to simply move the (virtual) disk to the new host and start the OSD, depending on the actual setup. If those are stand-alone OSDs (no separate DB/WAL) it shouldn't be too difficult [1]. If you're using ceph-volume you could run 'ceph-volume lvm trigger' in case the OSD doesn't restart. Although this would not reconstruct data it would start a rebalance process since the crushmap will change automatically, leading to misplaced objects. Regards, Eugen [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-March/025624.html Zitat von huxiaoyu(a)horebdata.cn: > Dear ceph folks, > > I encoutered an interesting situation as follows: an old FC SAN is > connected two ceph OSD nodes, and its LUNs are used as virtual OSDs. > When one node fails, its LUN can be taken over by anther node. My > question is, how to start up the OSD on the new node without > reconstructing its data? In other words, is there a simple to move > OSD from one node to another? > > many thanks, > > samuel > > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1