March 2020 - Dev - lists.ceph.io

by Patrick Donnelly

For developers submitting jobs using teuthology, we now have recommendations on what priority level to use: https://docs.ceph.com/docs/master/dev/developer_guide/#testing-priority -- Patrick Donnelly, Ph.D. He / Him / His Senior Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 months, 1 week

5
7
0 0

High iops on default.rgw.buckets.index

by Seena Fallah

Hi all. About 2 days my Ceph cluster goes 1 millions IO/s on reads default.rgw.buckets.index. When this happens my PUT requests goes up to 200 req/s but I had 500 req/s before and there were no high IO/s on that pool. When this happens my rgw nodes and OSDs goes up to 100% cpu usage. Do you have any indea what’s going on here that this pool gets 1 millions IO/s? Also I have upgraded to 14.2.8 but problem still persists. Thanks for you help :)

3 years

2
1
0 0

Windows port

by Lucian Petrut

Hi, We're happy to announce that a couple of weeks ago, we've submitted a few Github pull requests[1][2][3] adding initial Windows support. A big thank you to the people that have already reviewed the patches. To bring some context about the scope and current status of our work: we're mostly targeting the client side, allowing Windows hosts to consume rados, rbd and cephfs resources. We have Windows binaries capable of writing to rados pools[4]. We're using mingw to build the ceph components, mostly due to the fact that it requires the minimum amount of changes to cross compile ceph for Windows. However, we're soon going to switch to MSVC/Clang due to mingw limitations and long standing bugs[5][6]. Porting the unit tests is also something that we're currently working on. The next step will be implementing a virtual miniport driver so that RBD volumes can be exposed to Windows hosts and Hyper-V guests. We're hoping to leverage librbd as much as possible as part of a daemon that will communicate with the driver. We're also aiming at cephfs and considering using Dokan, which is FUSE compatible. Merging the open PRs would allow us to move forward, focusing on the drivers and avoiding rebase issues. Any help on that is greatly appreciated. Last but not least, I'd like to thank Suse, who's sponsoring this effort! Lucian Petrut Cloudbase Solutions [1] https://github.com/ceph/ceph/pull/31981 [2] https://github.com/ceph/ceph/pull/32027 [3] https://github.com/ceph/rocksdb/pull/42 [4] http://paste.openstack.org/raw/787534/ [5] https://sourceforge.net/p/mingw-w64/bugs/816/ [6] https://sourceforge.net/p/mingw-w64/bugs/527/

3 years

1
1
0 0

RGW -5 Error code

by Seena Fallah

Hi all. I often get this error on my PUT requests. rgw.log-552376-2020-03-23 15:12:53.270 7fe5aa6da700 0 WARNING: set_req_state_err err_no=5 resorting to 500 rgw.log-552377-2020-03-23 15:12:53.270 7fe5aa6da700 1 ====== req done req=0x7fe5aa6d38c0 op status=-5 http_status=500 latency=59.9965s ====== My put requests are always under 1s but this requests spend 1min! Any logs I can read to find out what's going on with my cluster?

3 years, 1 month

3
5
0 0

Resource requirements for integration test cluster

by Ulrich Weigand

Hello, we're currently investigating to set up a Teuthology cluster to run the Ceph integration test suite on IBM Z, to improve test coverage on our platform. However, we're not sure what hardware resources are required to do so. The target configuration should be large enough to comfortably support running an instance of the full Ceph integration tests. Is there some data available from your experience with such installations on how large this cluster needs to be then? In particular, what number of nodes, #cpus and memory per node, number (type/size) of disks that should be attached? Thanks for any data / estimates you can provide! Bye, Ulrich

3 years, 1 month

5
4
0 0

CDS day 2: Dashboard and orchestrator

by Sage Weil

Hi everyone, Thanks everyone who joined the session day on release process and testing. Unfortunately bluejeans had an issue halfway through the meeting that kicked everyone off temporarily.. and also lost the recording. I have a support ticket in to see if the recording is floating around somewhere but I imagine they're busy over there and won't get to it anytime soon. Tuesday we have two more sessions: 1100-1300 UTC: Dashboard https://pad.ceph.com/p/ceph-dashboard-pacific-priorities https://bluejeans.com/189591203 1400-1600 UTC: Orchestrator, cephadm, and rook https://pad.ceph.com/p/orchestrator-pacific https://bluejeans.com/822468990 As always, anyone is welcome to join the conversation and add topics to the agenda. Thanks! sage

3 years, 1 month

1
1
0 0

Does CephFS Fuse have to be slow?

by Robert LeBlanc

We have been evaluating other cluster storage solutions and one of them is just about as fast as Ceph, but only uses FUSE. They mentioned that recent improvement in the FUSE code allows for similar performance to kernel code. So, I'm doing some tests between CephFS kernel and FUSE and that is not true in the Ceph case. It seems that there is a lot of time spent in locks and polls. I'm wondering if this was needed to be done in the past to get around some deficiencies in FUSE, but are no longer needed. I don't know enough about FUSE to figure it out on my own. This is a very parallel workload running during these samples. Running `perf top`, I'm seeing: ``` 16.90% [kernel] [k] do_sys_poll 16.68% libopen-pal.so.20.10.1 [.] 0x0000000000082091 12.21% [kernel] [k] __fget 8.36% [kernel] [k] fput 7.01% [kernel] [k] tcp_poll 2.94% [kernel] [k] sock_poll 1.96% [vdso] [.] 0x0000000000000977 1.92% [kernel] [k] syscall_return_via_sysret 1.58% [kernel] [k] tcp_stream_memory_free ``` Annotating the do_sys_poll, I get ``` 0.09 │ → callq poll_freewait 0.09 │ mov -0x3d8(%rbp),%rcx │ lea -0x3b0(%rbp),%rsi │ xor %r8d,%r8d 0.00 │3f3: mov 0x8(%rsi),%eax 0.09 │ lea 0xc(%rsi),%r9 0.00 │ test %eax,%eax │ ↓ jle 4ce │ xor %edx,%edx │ ↓ jmp 416 2.03 │406: add $0x1,%edx 2.02 │ add $0x8,%rcx 6.33 │ cmp %edx,0x8(%rsi) 0.19 │ ↓ jle 4ce 0.09 │416: movslq %edx,%rax 1.99 │ movzwl 0x6(%r9,%rax,8),%edi 22.59 │ stac 2.01 │ mov %r8d,%eax 8.88 │ mov %di,0x6(%rcx) 26.62 │ clac 0.00 │ test %eax,%eax 2.12 │ ↑ je 406 │430: mov $0xfffffff2,%r13d 0.00 │436: mov -0x3b0(%rbp),%rdi │ test %rdi,%rdi 0.09 │ ↓ je 452 │442: mov (%rdi),%rbx │ → callq kfree │ test %rbx,%rbx │ mov %rbx,%rdi │ ↑ jne 442 ``` The libopen-pal.so.20.10.1 doesn't provide much info (because I'm not sure how to load the symbols) ``` 15.03% [.] 0x0000000000082091 0.62% [.] 0x0000000000082093 0.59% [.] opal_libevent2022_event_base_loop 0.50% [.] 0x00000000000820a0 0.47% [.] opal_progress 0.07% [.] 0x000000000006e41b 0.07% [.] opal_libevent2022_evutil_tv_to_msec ``` And in __fget ``` 2.52 │ sbb %rax,%rax 0.12 │ mov 0x8(%rdx),%rdx 0.14 │ and %edi,%eax 0.21 │ lea (%rdx,%rax,8),%rax 5.45 │ mov (%rax),%rdx 0.45 │ test %rdx,%rdx │ ↓ je 5c 19.25 │ test %esi,0x44(%rdx) │ ↓ jne 76 3.15 │ mov 0x38(%rdx),%rax 2.33 │ test %rax,%rax │ ↑ je 1c 0.00 │ lea 0x1(%rax),%rcx 0.19 │ lea 0x38(%rdx),%r10 58.09 │ lock cmpxchg %rcx,0x38(%rdx) 0.02 │ ↓ jne 61 2.31 │5c: mov %rdx,%rax 0.00 │ pop %rbp 0.00 │ ← retq ``` Thank you, ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 years, 1 month

4
6
0 0

Re: [ceph-users] Re: RGW failing to create bucket

by Casey Bodley

Any GSOC mentors available to help Abhinav here? On Thu, Mar 26, 2020 at 1:28 PM Abhinav Singh <singhabhinav0796(a)gmail.com> wrote: > > hey kind of help will be one step closure to solve this > > On Thu, 26 Mar 2020, 01:45 Abhinav Singh, <singhabhinav0796(a)gmail.com> wrote: >> >> I m gsoc student who is trying to figure it out..a little help would be great :) >> >> On Wed, 25 Mar 2020, 15:28 Abhinav Singh, <singhabhinav0796(a)gmail.com> wrote: >>> >>> please could you provide me some help a little bit of help will be of great use >>> >>> Thank you >>> >>> >>> On Tue, 24 Mar 2020, 23:13 Abhinav Singh, <singhabhinav0796(a)gmail.com> wrote: >>>> >>>> Yes I have checked the logs but the unable to understand what the error message, but unable to understand it. >>>> I m attaching it below please have a look. >>>> >>>> >>>> >>>> On Tue, Mar 24, 2020 at 5:02 PM Casey Bodley <cbodley(a)redhat.com> wrote: >>>>> >>>>> >>>>> >>>>> On Tue, Mar 24, 2020 at 6:14 AM Abhinav Singh <singhabhinav0796(a)gmail.com> wrote: >>>>>> >>>>>> anyone? >>>>>> >>>>>> On Mon, 23 Mar 2020, 23:39 Abhinav Singh, <singhabhinav0796(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>> > please someone help me >>>>>> > >>>>>> > On Mon, 23 Mar 2020, 19:44 Abhinav Singh, <singhabhinav0796(a)gmail.com> >>>>>> > wrote: >>>>>> > >>>>>> >> >>>>>> >> >>>>>> >> ---------- Forwarded message --------- >>>>>> >> From: Abhinav Singh <singhabhinav0796(a)gmail.com> >>>>>> >> Date: Mon, Mar 23, 2020 at 7:43 PM >>>>>> >> Subject: RGW failing to create bucket >>>>>> >> To: <dev(a)ceph.io> >>>>>> >> >>>>>> >> >>>>>> >> ceph : octopus >>>>>> >> JaegerTracing : master >>>>>> >> ubuntu : 18.04 >>>>>> >> >>>>>> >> When I implementing jaeger tracing it is unable to create a bucket. >>>>>> >> (I m using swif to perform testing.) >>>>>> >> /src/librados/IoCtxImpl.cc >>>>>> >> >>>>>> >> ``` >>>>>> >> void librados::IoCtxImpl::queue_aio_write(AioCompletionImpl *c) >>>>>> >> { >>>>>> >> std::cout<<"yes"<<std::endl; >>>>>> >> JTracer tracer; >>>>>> >> tracer.initTracer("Writing Started", >>>>>> >> "/home/abhinav/Desktop/GSOC/deepika/ceph/src/librados/tracerConfig.yaml" >>>>>> >> ); >>>>>> >> Span span=tracer.newSpan("writing started"); >>>>>> >> span->Finish(); >>>>>> >> try{ >>>>>> >> auto yaml = YAML::LoadFile("tracerConfig.yaml"); >>>>>> >> }catch(const YAML::ParserException& pe){ >>>>>> >> // ldout<<pe.what()<<dendl; >>>>>> >> std::cout<<pe.what()<<std::endl; >>>>>> >> ofstream f; >>>>>> >> f.open("/home/abhinav/Desktop/err.txt"); >>>>>> >> f<<pe.what(); >>>>>> >> f.close(); >>>>>> >> } >>>>>> >> // auto config = jaegertracing::Config::parse(yaml); >>>>>> >> // auto tracer=jaegertracing::Tracer::make( >>>>>> >> // "Writing", >>>>>> >> // config, >>>>>> >> // jaegertracing::logging::consoleLogger() >>>>>> >> // ); >>>>>> >> // opentracing::Tracer::InitGlobal( >>>>>> >> // static_pointer_cast<opentracing::Tracer>(tracer) >>>>>> >> // ); >>>>>> >> // auto span = opentracing::Tracer::Global()->StartSpan("Span1"); >>>>>> >> get(); >>>>>> >> ofstream file; >>>>>> >> file.open("/home/abhinav/Desktop/write.txt",std::ios::out | std::ios >>>>>> >> ::app); >>>>>> >> file<<"Writing /src/librados/IoCtxImpl.cc 310.\n"; >>>>>> >> file.close(); >>>>>> >> std::scoped_lock l{aio_write_list_lock}; >>>>>> >> ceph_assert(c->io == this); >>>>>> >> c->aio_write_seq = ++aio_write_seq; >>>>>> >> ldout(client->cct, 20) << "queue_aio_write " << this << " completion " >>>>>> >> << c >>>>>> >> << " write_seq " << aio_write_seq << dendl; >>>>>> >> aio_write_list.push_back(&c->aio_write_list_item); >>>>>> >> // opentracing::Tracer::Global()->Close(); >>>>>> >> } >>>>>> >> ``` >>>>>> >> /include/tracer.h >>>>>> >> ``` >>>>>> >> typedef std::unique_ptr<opentracing::Span> Span; >>>>>> >> >>>>>> >> class JTracer{ >>>>>> >> public: >>>>>> >> JTracer(){} >>>>>> >> ~JTracer(){ >>>>>> >> opentracing::Tracer::Global()->Close(); >>>>>> >> } >>>>>> >> void static inline loadYamlConfigFile(const char* path){ >>>>>> >> return; >>>>>> >> } >>>>>> >> void initTracer(const char* tracerName,const char* filePath){ >>>>>> >> auto yaml = YAML::LoadFile(filePath); >>>>>> >> auto configuration = jaegertracing::Config::parse(yaml); >>>>>> >> auto tracer = jaegertracing::Tracer::make( >>>>>> >> tracerName, >>>>>> >> configuration, >>>>>> >> jaegertracing::logging::consoleLogger()); >>>>>> >> opentracing::Tracer::InitGlobal( >>>>>> >> std::static_pointer_cast<opentracing::Tracer>(tracer)); >>>>>> >> Span s=opentracing::Tracer::Global()->StartSpan("Testing"); >>>>>> >> s->Finish(); >>>>>> >> } >>>>>> >> Span newSpan(const char* spanName){ >>>>>> >> Span span=opentracing::Tracer::Global()->StartSpan(spanName); >>>>>> >> return std::move(span); >>>>>> >> } >>>>>> >> Span childSpan(const char* spanName,const Span& parentSpan){ >>>>>> >> Span span = opentracing::Tracer::Global()->StartSpan(spanName, { >>>>>> >> opentracing::ChildOf(&parentSpan->context())}); >>>>>> >> return std::move(span); >>>>>> >> } >>>>>> >> Span followUpSpan(const char *spanName, const Span& parentSpan){ >>>>>> >> Span span = opentracing::Tracer::Global()->StartSpan(spanName, { >>>>>> >> opentracing::FollowsFrom(&parentSpan->context())}); >>>>>> >> return std::move(span); >>>>>> >> } >>>>>> >> }; >>>>>> >> ``` >>>>>> >> >>>>>> >> Output when trying to create new container >>>>>> >> >>>>>> >> ``` >>>>>> >> errno 111 connection refused >>>>>> >> ``` >>>>> >>>>> >>>>> connection refused probably means that radosgw crashed or isn't running. if it crashed, you might find out why by looking at its log file >>>>> >>>>>> >> But when I remove the tracer part in IoCtxImpl.cc it is workng fine. >>>>>> >> >>>>>> >> I m new to ceph, and dont know what information to share to correctly >>>>>> >> track down the problem, if any extra informtion is needed I will share it >>>>>> >> instantly. >>>>>> >> >>>>>> >> Been stuck into this issue for one week. >>>>>> >> Please someone help me! >>>>>> >> >>>>>> >> Thank you. >>>>>> >> >>>>>> > >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>

3 years, 1 month

2
1
0 0

Last set PRs for mimic point release

by Abhishek Lekshmanan

If anyone has PRs they need in the next mimic point, please tag them with mimic-batch-1 or reply in this thread, so that we can go for the final qe next. -- Abhishek

3 years, 1 month

3
3
0 0

CDS Pacific: release mgmt, teuthology session coming up!

by Sage Weil

In one hour, at 1400 UTC https://bluejeans.com/481913587 Agenda includes: - release management - hotfix releases - target OSes, architectures - git practices - branching - forward vs backporting - backport process - teuthology status and future (kyr) - py3 - gsoc topics

3 years, 1 month

1
0
0 0

2023

2022

2021

2020

2019

Dev March 2020