- ceph-users - lists.ceph.io

July Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 28 Jul '20

28 Jul '20

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday July 22nd. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200722 We try to keep it to an hour or less. Ceph calendar event details: July 22, 2020 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

2 1

repeatable crash in librbd1
by Johannes Naab 28 Jul '20

28 Jul '20

Hi, we observe crashes in librbd1 on specific workloads in virtual machines on Ubuntu 20.04 hosts with librbd1=15.2.4-1focal. The changes in https://github.com/ceph/ceph/commit/50694f790245ca90a3b8a644da7b128a7a148cc6 could be related, but do not easily apply against v15.2.4. We have collected several backtraces, using a reliable local reproducer. (This is a resent of the message stuck in the moderation queue since yesterday.) Best regards Johannes VM in libvirt with: <pre> <disk type='network' device='disk'> <driver name='qemu' type='raw' discard='unmap'/> <source protocol='rbd' name='pool/disk' index='4'>  </source> <iotune> <read_bytes_sec>209715200</read_bytes_sec> <write_bytes_sec>209715200</write_bytes_sec> <read_iops_sec>5000</read_iops_sec> <write_iops_sec>5000</write_iops_sec> <read_bytes_sec_max>314572800</read_bytes_sec_max> <write_bytes_sec_max>314572800</write_bytes_sec_max> <read_iops_sec_max>7500</read_iops_sec_max> <write_iops_sec_max>7500</write_iops_sec_max> <read_bytes_sec_max_length>60</read_bytes_sec_max_length> <write_bytes_sec_max_length>60</write_bytes_sec_max_length> <read_iops_sec_max_length>60</read_iops_sec_max_length> <write_iops_sec_max_length>60</write_iops_sec_max_length> </iotune> </disk> </pre> workload: <pre> fio --rw=write --name=test --size=10M timeout 30s fio --rw=write --name=test --size=20G timeout 3m fio --rw=write --name=test --size=20G --direct=1 timeout 1m fio --rw=randrw --name=test --size=20G --direct=1 timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1 # the backtraces are then observed while the following command is running fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test --size=1G --direct=1 </pre> observed stack traces three times: <pre> #0 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x557f633e9400) at ./src/common/event_socket.h:32 #1 0x00007ffb9740ba34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:262 #2 0x00007ffb9740ce98 in librbd::io::AioCompletion::complete (this=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:104 #3 0x00007ffb9740d1a0 in librbd::io::AioCompletion::complete_request (this=0x557f633f7600, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #4 0x00007ffb9742fdca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x7ffb68364a60, r=4096) at ./src/librbd/io/ReadResult.cc:155 #5 0x00007ffb9728334d in Context::complete (this=0x7ffb68364a60, r=<optimized out>) at ./src/include/Context.h:77 #6 0x00007ffb9742c7d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x7ffb6832bed0, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #7 0x00007ffb9742c735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #8 0x00007ffb9755e942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x7ffb683394a0, r=r@entry=0) at ./src/include/Context.h:78 #9 0x00007ffb97562e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x7ffb683394a0, r=0) at ./src/log/SubsystemMap.h:72 #10 0x00007ffb970f4177 in librados::C_AioComplete::finish (this=0x7ffb68362e30, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #11 0x00007ffb970afd1d in Context::complete (this=0x7ffb68362e30, r=<optimized out>) at ./src/include/Context.h:77 #12 0x00007ffb8765791d in Finisher::finisher_thread_entry (this=0x557f63387a30) at ./src/common/Finisher.cc:66 #13 0x00007ffb9946f609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #14 0x00007ffb99396103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> once: <pre> #0 0x00007fe7a972169d in std::atomic<boost::lockfree::detail::tagged_ptr<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::load (__m=std::memory_order_acquire, this=<optimized out>) at /usr/include/c++/9/atomic:250 #1 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:311 #2 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #3 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x5613db630440) at ./src/librbd/io/AioCompletion.cc:276 #4 0x00007fe7a9721a34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:262 #5 0x00007fe7a9722e98 in librbd::io::AioCompletion::complete (this=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:104 #6 0x00007fe7a97231a0 in librbd::io::AioCompletion::complete_request (this=0x5613dbbfa2c0, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #7 0x00007fe7a9745dca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x5613dc6a5c90, r=4096) at ./src/librbd/io/ReadResult.cc:155 #8 0x00007fe7a959934d in Context::complete (this=0x5613dc6a5c90, r=<optimized out>) at ./src/include/Context.h:77 #9 0x00007fe7a97427d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x5613db8bae40, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #10 0x00007fe7a9742735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #11 0x00007fe7a9874942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x5613dbbfc900, r=r@entry=0) at ./src/include/Context.h:78 #12 0x00007fe7a9878e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x5613dbbfc900, r=0) at ./src/log/SubsystemMap.h:72 #13 0x00007fe7a940a177 in librados::C_AioComplete::finish (this=0x5613db85e6d0, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #14 0x00007fe7a93c5d1d in Context::complete (this=0x5613db85e6d0, r=<optimized out>) at ./src/include/Context.h:77 #15 0x00007fe79b65791d in Finisher::finisher_thread_entry (this=0x5613db369ee0) at ./src/common/Finisher.cc:66 #16 0x00007fe7ab788609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #17 0x00007fe7ab6af103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> twice: <pre> #0 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::extract_ptr (i=<error reading variable>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:113 #1 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::get_ptr (this=0xfffff40ee9c0) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:115 #2 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate_impl<false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:187 #3 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate<true, false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:168 #4 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::construct<true, false, librbd::io::AioCompletion*, boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node*> (arg2=<optimized out>, arg1=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:100 #5 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:302 #6 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #7 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x560f5e10c000) at ./src/librbd/io/AioCompletion.cc:276 #8 0x00007f8c8bcada34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x560f5edd5d30) at ./src/librbd/io/AioCompletion.cc:262 #9 0x00007f8c8bcaee98 in librbd::io::AioCompletion::complete (this=0x560f5edd5d30) at ./src/librbd/io/AioCompletion.cc:104 #10 0x00007f8c8bcaf1a0 in librbd::io::AioCompletion::complete_request (this=0x560f5edd5d30, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #11 0x00007f8c8bcd1dca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x7f8bc4342280, r=4096) at ./src/librbd/io/ReadResult.cc:155 #12 0x00007f8c8bb2534d in Context::complete (this=0x7f8bc4342280, r=<optimized out>) at ./src/include/Context.h:77 #13 0x00007f8c8bcce7d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x7f8bc4326820, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #14 0x00007f8c8bcce735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #15 0x00007f8c8be00942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x7f8bc42b2a30, r=r@entry=0) at ./src/include/Context.h:78 #16 0x00007f8c8be04e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x7f8bc42b2a30, r=0) at ./src/log/SubsystemMap.h:72 #17 0x00007f8c8b996177 in librados::C_AioComplete::finish (this=0x7f8bc43423b0, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #18 0x00007f8c8b951d1d in Context::complete (this=0x7f8bc43423b0, r=<optimized out>) at ./src/include/Context.h:77 #19 0x00007f8c82f4291d in Finisher::finisher_thread_entry (this=0x560f5f44b180) at ./src/common/Finisher.cc:66 #20 0x00007f8c91e1a609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #21 0x00007f8c91d41103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> once: <pre> #0 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::extract_ptr (i=<error reading variable>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:113 #1 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::get_ptr (this=0xfffff40ee9c0) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:115 #2 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate_impl<false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:187 #3 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate<true, false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:168 #4 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::construct<true, false, librbd::io::AioCompletion*, boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node*> (arg2=<optimized out>, arg1=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:100 #5 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:302 #6 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #7 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x55d1b89113c0) at ./src/librbd/io/AioCompletion.cc:276 #8 0x00007f275f6d7a34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x55d1b8a7a0d0) at ./src/librbd/io/AioCompletion.cc:262 #9 0x00007f275f6d8e98 in librbd::io::AioCompletion::complete (this=0x55d1b8a7a0d0) at ./src/librbd/io/AioCompletion.cc:104 #10 0x00007f275f6d91a0 in librbd::io::AioCompletion::complete_request (this=0x55d1b8a7a0d0, r=<optimized out>) at ./src/librbd/io/AioCompletion.cc:229 #11 0x00007f275f54f34d in Context::complete (this=0x7f268833f850, r=<optimized out>) at ./src/include/Context.h:77 #12 0x00007f274f6b647b in ThreadPool::worker (this=0x55d1b9d04860, wt=<optimized out>) at ./src/common/WorkQueue.cc:118 #13 0x00007f274f6b7545 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:466 #14 0x00007f276173e609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #15 0x00007f2761665103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre>

2 6

slow ops on one osd makes all my buckets unavailable
by Zhenshi Zhou 28 Jul '20

28 Jul '20

Hi, My harbor registry uses ceph object storage to save the images. But I couldn't pull/push images from harbor a few moments ago. Ceph was in warning health status in the same time. The cluster just had a warning message said that osd.24 has slow ops. I check the ceph-osd.24.log, and showed as below: *2020-07-28 19:01:40.599 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:41.558 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:42.579 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:43.566 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:44.588 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:45.627 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:46.674 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:47.701 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:48.729 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:49.729 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:50.889 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)......* *......* *2020-07-28 21:03:35.053 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.166298.0:34904067 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)* After restarted osd.24, the cluster became health again, so was harbor. What confuse me is that why my harbor couldn't get data from its bucket while the log indicated that there was a client block on the other bucket. I don't think some slow ops on an osd has any bad effect on all buckets. Any idea is appreciated. Thanks

1 0

S3 bucket lifecycle not deleting old objects
by Alex Hussein-Kershaw 28 Jul '20

28 Jul '20

Hello, I have a problem that old versions of S3 objects are not being deleted. Can anyone advise as to why? I'm using Ceph 14.2.9. I expect old versions of S3 objects to be deleted after 3 days as per my lifecycle config on the bucket: { "Rules": [ { "Status": "Enabled", "Prefix": "", "NoncurrentVersionExpiration": { "NoncurrentDays": 3 }, "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "S3 scsdata bucket: Tidy up old versions" } ] } I have an object with 3 versions below (and is much older than 3 days): [root@hera hera_sdc] /usr/bin> aws s3api --endpoint http://127.3.3.3:7480 list-object-versions --bucket hera-scsdata --key 84/46/2020060508501821902143658709-Subscriber | grep -B 4 -A 6 84/46/2020060508501821902143658709-Subscriber "LastModified": "2020-06-05T08:58:19.644Z", "VersionId": "FUdZIehBu3sgRbNJSmZwj3VHWs1ednH", "ETag": "\"a18286c50a7323efe58497eb97d6dc9d\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": true, "Size": 4440 -- "LastModified": "2020-06-05T08:58:17.943Z", "VersionId": "JVKGMJQS-l7xKQuqdfn4QsEY5WLEosj", "ETag": "\"87e9953af436b702afb80d457f1d73cb\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": false, "Size": 4408 -- "LastModified": "2020-06-05T08:50:19.167Z", "VersionId": "-RSNSCDvGj83f4DZ11s8YZ2KaxT8T.a", "ETag": "\"a68ec68ce825e009ee9a70cfdae9c794\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": false, "Size": 4256 -- ], "NextKeyMarker": "85/49/20200604163626B4C712312312302641-Subscriber", "MaxKeys": 1000, "Prefix": "", "KeyMarker": "84/46/2020060508501821902143658709-Subscriber", "DeleteMarkers": [ { "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, So those objects still being present seems to be in conflict with the config I have set? Thanks, Alex

1 0

Re: Push config to all hosts
by Cem Zafer 28 Jul '20

28 Jul '20

Thanks Ricardo for clarification. Regards. On Mon, Jul 27, 2020 at 2:50 PM Ricardo Marques <RiMarques(a)suse.com> wrote: > Hi Cem, > > Since https://github.com/ceph/ceph/pull/35576 you will be able to tell > cephadm to keep your `/etc/ceph/ceph.conf` updated in all hosts by runnig: > > # ceph config set mgr mgr/cephadm/manage_etc_ceph_ceph_conf true > > But this feature was not released yet, so you will have to wait for > v15.2.5. > > > Ricardo Marques > > ------------------------------ > *From:* Cem Zafer <cemzafer(a)gmail.com> > *Sent:* Monday, June 29, 2020 6:37 AM > *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Push config to all hosts > > Hi, > What is the best method(s) to push ceph.conf to all hosts in octopus > (15.x)? > Thanks. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Weird buckets in a new cluster causing broken dashboard functionality
by Eugen König 28 Jul '20

28 Jul '20

Hi all, I'm currently experience some strange behavior in our cluster: the dashboards object gateway "buckets" submenu is broken and I'm getting 503 errors (however, "Users" and "Daemons" work flawlessly). Looking into the mgr log gives me following error: 2020-07-24T12:38:12.695+0200 7f42150f3700 0 [dashboard ERROR rest_client] RGW REST API failed GET req status: 404 2020-07-24T12:38:12.843+0200 7f42130ef700 0 [dashboard ERROR request] [10.1.0.133:38454] [GET] [500] [0.039s] [eugen] [513.0B] /api/rgw/bucket/91e22800581543b5be4654f7b9b0c7cc_102020-07-24T12:38:12.843+0200 7f42130ef700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "00a3c78a-d96f-4f8b-b3c4-f24eac99f4a1"} '] So it took me to the point to look into buckets and I got this list with weird bucket names (even with dash, but it's actually not allowed): root@ceph1-40-10~# radosgw-admin buckets list [ "12345", "deployment", "91e22800581543b5be4654f7b9b0c7cc_5", "91e22800581543b5be4654f7b9b0c7cc_6", "91e22800581543b5be4654f7b9b0c7cc_11", "91e22800581543b5be4654f7b9b0c7cc_8", "91e22800581543b5be4654f7b9b0c7cc_3", "91e22800581543b5be4654f7b9b0c7cc_17", [...] ] If I try to perform some operations on these buckets, I get an error: root@ceph1-40-10~# radosgw-admin bucket rm --bucket=91e22800581543b5be4654f7b9b0c7cc_15 --purge-objects 2020-07-28T12:40:05.392+0200 7f500024a080 -1 ERROR: unable to remove bucket(2) No such file or directory I'm able to change the owner and even rename it, but anything else is ending in the same error above. This also seems to break the buckets functionality in the dashboard. My question is: what are those buckets, where do they come from and what is stored in there? And why one is not able to perform any ops on them? Any ideas? Best, eugen

1 0

Re: please help me fix iSCSI Targets not available
by David Thuong 28 Jul '20

28 Jul '20

i user ceph octopus v15

1 0

HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error
by Fabrizio Cuseo 28 Jul '20

28 Jul '20

Hello, I use ceph with proxmox, release 14.2.9, with bluestore OSD. I had a problem in a replica 2 pool (i know, is dangerous), with an unexpected clone. I have deleted the RBD image that used the damaged object (with his snapshot), and now ceph can't trim and clean. How can I restore the clean and health status ? Regards, Fabrizio

2 1

Re: Cluster became unresponsive: e5 handle_auth_request failed to assign global_id
by Илья Борисович Волошин 28 Jul '20

28 Jul '20

No, they are stored locally on ESXi data storage on top of hardware RAID5 built with SAS/SATA (different hardware on hosts). Also, I've tried going back to the snapshot taken just after all monitors and OSDs were added to cluster. The host boots fine and is working as it should, however, after the next reboot this problem appears (no changes to configuration were made). And another thing - even if docker container for mgr is running and gives no errors in logs neither inside the container nor on the parent host the mgr doesn't bind to any ports it should: 6800, 6801 and 8443 for dashboard. Not sure if it is the reason or the consequence of this problem. вт, 28 июл. 2020 г. в 11:37, Anthony D'Atri <anthony.datri(a)gmail.com>: > Are your mon DBs on SSDs? > > > On Jul 27, 2020, at 7:28 AM, Илья Борисович Волошин < > i.voloshin(a)simplesolution.pro> wrote: > > > > Here are all the active ports on mon1 (with the exception of sshd and > ntpd): > > > > # netstat -npl > > Proto Recv-Q Send-Q Local Address Foreign Address State > > PID/Program name > > tcp 0 0 <mon1_ip>:3300 0.0.0.0:* LISTEN > > 1582/ceph-mon > > tcp 0 0 <mon1_ip>:6789 0.0.0.0:* > LISTEN > > 1582/ceph-mon > > tcp6 0 0 :::9093 :::* > LISTEN > > 908/alertmanager > > tcp6 0 0 :::9094 :::* > LISTEN > > 908/alertmanager > > tcp6 0 0 :::9095 :::* > LISTEN > > 896/prometheus > > tcp6 0 0 :::9100 :::* > LISTEN > > 906/node_exporter > > tcp6 0 0 :::3000 :::* > LISTEN > > 882/grafana-server > > udp6 0 0 :::9094 :::* > > 908/alertmanager > > > > I've tried telnet from mon1 host, can connect to 3300 and 6789: > > > > # telnet <mon1_ip> 3300 > > > Trying <mon1_ip>... > > Connected to <mon1_ip>. > > Escape character is '^]'. > > ceph v2 > > > > # telnet <mon1_ip> 6789 > > Trying <mon1_ip>... > > Connected to <mon1_ip>. > > Escape character is '^]'. > > ceph v027QQ > > > > 6800 and 6801 refuse connection: > > > > # telnet <mon1_ip> 6800 > > Trying <mon1_ip>... > > telnet: Unable to connect to remote host: Connection refused > > > > I don't see any errors in the log related to failures to bind... and all > > CEPH systemd services are running as far as I can tell: > > > > # systemctl list-units -a | grep ceph > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)alertmanager.mon1.service > > loaded active running Ceph > > alertmanager.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)crash.mon1.service > > loaded active running Ceph crash.mon1 > > for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)grafana.mon1.service > > loaded active running Ceph > grafana.mon1 > > for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)mgr.mon1.peevkl.service > > loaded active running Ceph > > mgr.mon1.peevkl for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)mon.mon1.service > > loaded active running Ceph mon.mon1 > for > > e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)node-exporter.mon1.service > > loaded active running Ceph > > node-exporter.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5(a)prometheus.mon1.service > > loaded active running Ceph > > prometheus.mon1 for e30397f0-cc32-11ea-8c8e-000c29469cd5 > > system-ceph\x2de30397f0\x2dcc32\x2d11ea\x2d8c8e\x2d000c29469cd5.slice > > loaded active active > > system-ceph\x2de30397f0\x2dcc32\x2d11ea\x2d8c8e\x2d000c29469cd5.slice > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5.target > > loaded active active Ceph cluster > > e30397f0-cc32-11ea-8c8e-000c29469cd5 > > ceph.target > > loaded active active All Ceph clusters > > and services > > > > Here are currently active docker images: > > > > # docker ps > > CONTAINER ID IMAGE COMMAND > > CREATED STATUS PORTS NAMES > > dfd8dbeccf1e ceph/ceph:v15 "/usr/bin/ceph-mgr -…" > > 41 minutes ago Up 41 minutes > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-mgr.mon1.peevkl > > 9452d1db7ffb ceph/ceph:v15 "/usr/bin/ceph-mon -…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-mon.mon1 > > 703ec4a43824 prom/prometheus:v2.18.1 "/bin/prometheus --c…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-prometheus.mon1 > > d816ec5e645f ceph/ceph:v15 "/usr/bin/ceph-crash…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-crash.mon1 > > 38d283ba6424 ceph/ceph-grafana:latest "/bin/sh -c 'grafana…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-grafana.mon1 > > cc119ec8f09a prom/node-exporter:v0.18.1 "/bin/node_exporter …" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-node-exporter.mon1 > > aa1d339c4100 prom/alertmanager:v0.20.0 "/bin/alertmanager -…" > 3 > > hours ago Up 3 hours > > ceph-e30397f0-cc32-11ea-8c8e-000c29469cd5-alertmanager.mon1 > > > > iptables are active, I tried setting all chain policies to ACCEPT (didn't > > help), the rules are as such: > > > > 0 0 CEPH tcp -- * * 0.0.0.0/0 > > 0.0.0.0/0 tcp dpt:6789 > > 5060 303K CEPH tcp -- * * 0.0.0.0/0 > > 0.0.0.0/0 multiport dports 6800:7300 > > > > Chain CEPH includes addresses for monitors and OSDs. > > > > пн, 27 июл. 2020 г. в 17:07, Dino Godor <dg(a)terralink.de>: > > > >> Hi, > >> > >> have you tried to locally connect to the ports with netcat (or telnet)? > >> > >> Is the process listening ? (something like netstat -4ln or the current > >> equivalent thereof) > >> > >> Is the old (new) Firewall maybe still running ? > >> > >> > >> On 27.07.20 16:00, Илья Борисович Волошин wrote: > >>> Hello, > >>> > >>> I've created an Octopus 15.2.4 cluster with 3 monitors and 3 OSDs (6 > >> hosts > >>> in total, all ESXi VMs). It lived through a couple of reboots without > >>> problem, then I've reconfigured the main host a bit: > >>> set iptables-legacy as current option in update-alternatives (this is a > >>> Debian10 system), applied a basic ruleset of iptables and restarted > >> docker. > >>> > >>> After that the cluster became unresponsive (any ceph command hangs > >>> indefinitely). I can use admin socket to manipulate config though. > >> Setting > >>> debug_ms to 5 I see this in the logs (timestamps cut for readability): > >>> > >>> 7f4096f41700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> [v2:<mon2_ip>:3300/0,v1:<mon2_ip>:6789/0] conn(0x55c21b975800 > >>> 0x55c21ab45180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx= > >>> 0).send_message enqueueing message m=0x55c21bd84a00 type=67 > >> mon_probe(probe > >>> e30397f0-cc32-11ea-8c8e-000c29469cd5 name mon1 mon_release octopus) v7 > >>> 7f4098744700 1 -- >> > >>> [v2:<mon1_ip>:6800/561959008,v1:<mon1_ip>:6801/561959008] > >>> conn(0x55c21b974400 msgr2=0x55c21ab45600 unknown :-1 > >> s=STATE_CONNECTING_RE > >>> l=0).process reconnect failed to v2:81.200.2 > >>> .152:6800/561959008 > >>> 7f4098744700 2 -- >> > >>> [v2:<mon1_ip>:6800/561959008,v1:<mon1_ip>:6801/561959008] > >>> conn(0x55c21b974400 msgr2=0x55c21ab45600 unknown :-1 > >> s=STATE_CONNECTING_RE > >>> l=0).process connection refused! > >>> > >>> and this: > >>> > >>> 7f4098744700 2 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0)._fault on lossy channel, failing > >>> 7f4098744700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0).stop > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0).reset_recv_state > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21ba38c00 0x55c21bcc5a80 secure :-1 s=AUTH_ACCEPTING pgs=0 > >> cs=0 > >>> l=1 rx=0 tx=0).reset_security > >>> 7f409373a700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 > >> rx=0 > >>> tx=0).accept > >>> 7f4098744700 1 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=BANNER_ACCEPTING > pgs=0 > >>> cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=HELLO_ACCEPTING pgs=0 > >>> cs=0 l=0 rx=0 tx=0).handle_hello received hello: peer_type=8 > >>> peer_addr_for_me=v2:<mon1_ip>:3300/0 > >>> 7f4098744700 5 --2- [v2:<mon1_ip>:3300/0,v1:<mon1_ip>:6789/0] >> > >>> conn(0x55c21c0d2800 0x55c21bcc3f80 unknown :-1 s=HELLO_ACCEPTING pgs=0 > >>> cs=0 l=0 rx=0 tx=0).handle_hello getsockname says I am <mon1_ip>:3300 > >> when > >>> talking to v2:<mon1_ip>:49012/0 > >>> 7f4098744700 1 mon.mon1@0(probing) e5 handle_auth_request failed to > >> assign > >>> global_id > >>> > >>> Config (the result of ceph --admin-daemon > >>> /run/ceph/e30397f0-cc32-11ea-8c8e-000c29469cd5/ceph-mon.mon1.asok > config > >>> show): > >>> https://pastebin.com/kifMXs9H > >>> > >>> I can connect to ports 3300 and 6789 with telnet; 6800 and 6801 return > >>> 'process connection refused' > >>> > >>> Setting all iptables policies to ACCEPT didn't change anything. > >>> > >>> Where should I start digging to fix this problem? I'd like to at least > >>> understand why this happened before putting the cluster into > production. > >>> Any help is appreciated. > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Server error when trying to view this list in browser
by biohazd＠yahoo.com 28 Jul '20

28 Jul '20

Hi I often get "Server error - An error occurred while processing your request." when tryignt o view this lis in Fireofx, is it a known issues? https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/new

1 0