September 2020 - ceph-users

by Robert Sander

Hi, yes, I have read https://docs.ceph.com/docs/octopus/cephadm/stability/ and know that the iSCSI support is still under development. But why is the command "ceph orch apply iscsi" available when cephadmin is the orchestrator backend? I am at the stage that I successfully rolled out an iSCSI gateway which is visible in the Ceph dashboard. But when trying to setup a target an error occurs: Sep 04 15:15:58 ceph02 rbd-target-api[124040]: Unable to create the Target definition - Could not load module: iscsi_target_mod Sep 04 15:15:58 ceph02 rbd-target-api[124040]: Unhandled Exception Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/rtslib_fb/node.py", line 71, in _create_in_cfs_ine os.mkdir(self.path) FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/config/target/iscsi' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/rtslib_fb/fabric.py", line 156, in _check_self self._create_in_cfs_ine('any') File "/usr/lib/python3.6/site-packages/rtslib_fb/node.py", line 74, in _create_in_cfs_ine % self.__class__.__name__) rtslib_fb.utils.RTSLibError: Could not create ISCSIFabricModule in configFS During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 432, in modprobe kmod.Kmod().modprobe(module) File "kmod/kmod.pyx", line 106, in kmod.kmod.Kmod.modprobe File "kmod/kmod.pyx", line 82, in lookup kmod.error.KmodError: Could not modprobe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/bin/rbd-target-api", line 106, in decorated return f(*args, **kwargs) File "/usr/bin/rbd-target-api", line 304, in target target.manage('init') File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/target.py", line 710, in manage 'mutual_password_encryption_enabled']) File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/discovery.py", line 14, in set_discovery_auth_lio iscsi_fabric.clear_discovery_auth_settings() File "/usr/lib/python3.6/site-packages/rtslib_fb/fabric.py", line 224, in clear_discovery_auth_settings self._check_self() File "/usr/lib/python3.6/site-packages/rtslib_fb/fabric.py", line 158, in _check_self modprobe(self.kernel_module) File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 434, in modprobe raise RTSLibError("Could not load module: %s" % module) rtslib_fb.utils.RTSLibError: Could not load module: iscsi_target_mod Solution: "modprobe iscsi_target_mod" on the host itself, the container is not allowed to do that. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 years, 7 months

3
2
0 0

how to reduce osd down interval on laggy disk ?

by Alexandre DERUMIER

Hi, I have sometime laggy ssd drives (intel s3610), maybe a firmware bug or controller bug. (I have tested with differents firmware, don't help). When this occur, I have this kind of error Aug 29 15:43:37 ceph5-7 kernel: [447163.801090] sd 0:0:3:0: Power-on or device reset occurred Just before this, the disk is lagging for 1-2min, and iowait increase a lot. Then monitor is mark it down. (at almost the same time than the disk reset, not sure it's related) Is it possible to reduce the threshold to force the osd down faster ? (maybe 30s for example). (I'm running ceph nautilus 14.2.6) The osd log is: 020-08-29 15:43:37.610 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5618a5f27200, latency = 93.062018 2020-08-29 15:43:37.610 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5618a2956c00, latency = 93.031658 2020-08-29 15:43:37.610 7fa1025b7700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x10543123000~1000 (direct) since 447078s, timeout is 5s 2020-08-29 15:43:37.610 7fa1025b7700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 93.0167s, num_ios = 3072 2020-08-29 15:43:37.642 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56197c0f1200, latency = 92.892911 2020-08-29 15:43:37.642 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56194151db00, latency = 82.923060 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56197c0f0900, latency = 92.611975 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56190ea76000, latency = 92.439439 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x561964172000, latency = 92.861119 2020-08-29 15:43:37.646 7fa0fedb0700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0xf10fa5c000~1000 (direct) since 447091s, timeout is 5s 2020-08-29 15:43:37.646 7fa0fd5ad700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0xe08a1000~1000 (direct) since 447079s, timeout is 5s 2020-08-29 15:43:37.646 7fa0fedb0700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 79.5633s, num_ios = 2048 2020-08-29 15:43:37.646 7fa0fd5ad700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 92.3464s, num_ios = 18 2020-08-29 15:43:37.646 7fa1045bb700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x103a3da1000~1000 (direct) since 447078s, timeout is 5s 2020-08-29 15:43:37.646 7fa0ffdb2700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x100a4d26000~1000 (direct) since 447078s, timeout is 5s 2020-08-29 15:43:37.646 7fa0ffdb2700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 93.022s, num_ios = 1024 2020-08-29 15:43:37.646 7fa1045bb700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 92.5954s, num_ios = 1536 2020-08-29 15:43:37.646 7fa1005b3700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x1003a505000~1000 (direct) since 447079s, timeout is 5s 2020-08-29 15:43:37.646 7fa101db6700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0x1012ea45000~1000 (direct) since 447080s, timeout is 5s 2020-08-29 15:43:37.646 7fa1005b3700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 92.3891s, num_ios = 3584 2020-08-29 15:43:37.646 7fa101db6700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 91.1452s, num_ios = 2048 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56191dbce300, latency = 82.924723 2020-08-29 15:43:37.646 7fa0fcdac700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) read stalled read 0xfe7a8b7000~1000 (direct) since 447083s, timeout is 5s 2020-08-29 15:43:37.646 7fa0fcdac700 0 bluestore(/var/lib/ceph/osd/ceph-43) log_latency_fn slow operation observed for _do_read, latency = 87.9135s, num_ios = 512 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x56196a946300, latency = 82.923270 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5618c70d8f00, latency = 92.344641 2020-08-29 15:43:37.646 7fa129e17700 0 bluestore(/var/lib/ceph/osd/ceph-43) _txc_state_proc slow aio_wait, txc = 0x5619748d1500, latency = 92.333448 d for _txc_committed_kv, latency = 91.2568s, txc = 0x56196ab26f00 2020-08-29 15:43:37.774 7fa10e5cf700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.43 down, but it is still running ..... ..... 2020-08-29 15:43:37.774 7fa10e5cf700 0 log_channel(cluster) log [DBG] : map e604017 wrongly marked me down at e604016 2020-08-29 15:43:39.494 7fa0fd5ad700 -1 bdev(0x5618a06c4000 /var/lib/ceph/osd/ceph-43/block) aio_submit retries 15 monitor logs --------------- mon1 ---- 2020-08-29 15:43:09.836 7f3ed3030700 0 log_channel(audit) log [DBG] : from='client.835295911 X.X.0.43:0/3364930077' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch 2020-08-29 15:43:10.140 7f3ed5835700 0 log_channel(cluster) log [WRN] : Health check update: 715 slow ops, oldest one blocked for 62 sec, daemons [osd.13,osd.14,osd.15,osd.16,osd.17,osd.20,osd.22,osd.23,osd.26,osd.27]... have slow ops. (SLOW_OPS) mon2 ---- 2020-08-29 15:43:20.140 7f3ed5835700 0 log_channel(cluster) log [WRN] : Health check update: 817 slow ops, oldest one blocked for 73 sec, daemons [osd.13,osd.14,osd.15,osd.16,osd.17,osd.20,osd.22,osd.23,osd.26,osd.27]... have slow ops. (SLOW_OPS) mon3 ------ 2020-08-29 15:43:08.066 7f3d213c5700 0 mon.ceph5-2@1(peon) e5 handle_command mon_command({"prefix": "osd blacklist", "blacklistop": "add", "addr": "10.5.0.93:0/955438826"} v 0) v1 2020-08-29 15:43:08.066 7f3d213c5700 0 log_channel(audit) log [INF] : from='client.835295776 X.X.0.92:0/2918303586' entity='client.admin' cmd=[{"prefix": "osd blacklist", "blacklistop": "add", "addr": "10.5.0.93:0/955438826"}]: dispatch 2020-08-29 15:43:09.974 7f3d213c5700 0 mon.ceph5-2@1(peon) e5 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1 2020-08-29 15:43:09.974 7f3d213c5700 0 log_channel(audit) log [DBG] : from='client.? X.X.0.44:0/728500154' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch 2020-08-29 15:43:10.330 7f3d213c5700 0 mon.ceph5-2@1(peon) e5 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1 2020-08-29 15:43

3 years, 7 months

1
0
0 0

Multipart upload issue from Java SDK clients

by shubjero

Good day, I am having an issue with some multipart uploads to radosgw. I recently upgraded my cluster from Mimic to Nautilus and began having problems with multipart uploads from clients using the Java AWS SDK (specifically 1.11.219). I do NOT have issues with multipart uploads with other clients such as the aws2 cli. The error that surfaces is http error 500. Small files that are not multipart uploads work fine with the Java client. I enabled debugging for debug_ms = 1 and debug_rgw = 20 and then performed a multipart upload with the offending client and this is what I see (full log @ https://object.cancercollaboratory.org:9080/swift/v1/ceph-debug-pub/filtere… ) - just a snippet - 2020-09-02 18:03:43.339 7f9277474700 1 -- 172.25.13.93:0/458188962 --> [v2:172.25.12.152:6936/12180,v1:172.25.12.152:6937/12180] -- osd_op(unknown.0.0:6331 17.325 17:a4c2f681:::2020-09-02-18-default.262831163.2-score-rdpc-collab-qa:head [append 0~817] snapc 0=[] ondisk+write+known_if_redirected e1493993) v8 -- 0x55e9fefb9000 con 0x55e9f3e61f80 2020-09-02 18:03:43.339 7f9277474700 2 req 1714 70.432s s3:put_obj op status=-35 2020-09-02 18:03:43.339 7f9277474700 2 req 1714 70.432s s3:put_obj http status=500 haproxy log: Sep 2 13:33:36 localhost haproxy[14495]: 206.108.127.16:46088 [02/Sep/2020:13:32:06.263] radosgw_public_https_11~ radosgw_public_https_11/storage2-r5 0/0/0/71677/89948 500 477 - - ---- 12/12/11/1/0 0/0 {} "PUT /score-rdpc-collab-qa/data/1c9e4395-677c-55c1-8437-b6467c806173?uploadId=2~zTiBm6p4sB2SyMu2cJ_dwd3ECMpELh3&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20200902T173202Z&X-Amz-SignedHeaders=host&X-Amz-Expires=518400&X-Amz-Credential=TO1DRTS4H14HJ388I26R%2F20200902%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=225b6f111cc342c97ad48f8a39f34af044647ac04da532ef949d770509e65f76 HTTP/1.1" Does anyone know if there may be incompatiblities with Java AWS SDK v1? I see that v2 is out so one of our devs is going to run some tests using Java AWS SDK v2 but this may take a bit of time. Thanks all, Jared Baker Ontario Institute for Cancer Research

3 years, 7 months

1
1
0 0

Change Roadrunner Email Password | RR Email Help

by elouisesng5445＠gmail.com

Roadrunner Recovery by Roadrunner Email Password is a easy process. We have often forgotten our new password because we don't want it, or it's always stolen because of account. So the network is designed to overcome these Roadrunner email authentication failures, where we can find a path to our solution. When you Change Roadrunner Password you don't have to worry at all, because resetting it is very fast. You will find the full description of the Change Roadrunner password here, in this article. Also we'll send you some tips and tricks to keep your Roadrunner email account safe. for More Information Please Visit: http://www.emailscustomercare.com/change-roadrunner-password/

3 years, 7 months

1
0
0 0

International B2B Marketplace

by Grace Stefan

Exporthub is an international <a href="https://www.exporthub.com/">b2b marketplace</a> catering to thousands of globally active manufacturers & suppliers from all industries with its safest trade services. Exporthub is dedicated to making international trading more comfortable than ever. Our team of experts is keen to provide all global buyers and suppliers with tools to help them succeed in their particular industry. Our range of services allows suppliers to connect with buyers worldwide for their products. As a B2B trade portal, we provide a B2B platform that lets all American, Indian, African, and Chinese B2B site to enhance their network and connect with businesses across their borders.

3 years, 7 months

1
0
1 0

Did I use Cash.app/help to utilize it without telephone number?

by mary smith

You don't have a telephone number and you attempted to sign in to your Cash app account. Most importantly, click on the profile symbol and afterward select the Personal tab. Presently, you have to include any missing email address or telephone number. It fundamentally needs to get an affirmation code. Hello, click here and get Cash.app/help for any issues. https://www.cashapp-customerservice.com/

3 years, 7 months

1
0
0 0

Actual block size of osd

by Ragan, Tj (Dr.)

Hi All, Does anyone know how to get the actual block size used by an osd? I’m trying to evaluate 4k vs 64k min_alloc_size_hdd and want to verify that the newly created osds are actually using the expected block size. Thanks, -TJ Ragan

3 years, 7 months

2
1
0 0

Re: Change fsid of Ceph cluster after splitting it into two clusters

by Dan van der Ster

Hi Wido, Out of curiosity, did you ever work out how to do this? Cheers, Dan On Tue, Feb 12, 2019 at 6:17 PM Wido den Hollander <wido(a)42on.com> wrote: > > Hi, > > I've got a situation where I need to split a Ceph cluster into two. > > This cluster is currently running a mix of RBD and RGW and in this case > I am splitting it into two different clusters. > > A difficult thing to do, but it's possible. > > One problem that stays though is that after the split both Ceph clusters > have the same fsid and that might be confusing. > > Is there a way to change the fsid of an existing cluster? > > Injecting an updated MONMAP and OSDMAP into the cluster? > > It's no problem if this has to be done offline, but I'm just wondering > if this is possible. > > Wido > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

3 years, 7 months

2
1
0 0

Is it possible to change the cluster network on a production ceph?

by psousa＠alticelabs.com

Hello people, I am trying to change the cluster network in a production ceph. I'm having problems, after changing the ceph.conf file and restarting a osd the cluster is always going to HEALTH_ERROR with blocked requests. Only by returning to the previous configuration and restarting the same osd make the cluster going to OK. So, my question is: is it possible to change the cluster network on a production ceph without stopping the service and destroying data? If so, how? Anticipated thanks Paulo Sousa

3 years, 7 months

2
1
0 0

Cheapest essay writing service

by omeojones＠gmail.com

One of the unique features of our portal is our 24/7 live chat support platform. Students from any USA location can use this facility to directly receive essay assignment tips from our easy essay writer. https://myassignmenthelp.com/us/cheap-essay-writing-services.html

3 years, 7 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users September 2020