- ceph-users - lists.ceph.io

by hirkumispe＠enayu.com

Fix technical breakdown of all your electronics and appliances at Geek Squad Support. Reach the certified experts at Geek Squad Support for fixing any kind of technical bug with your devices. Best of services and assistance assured at support. https://geekstechs.org/geek-squad-support/ Remove the technical glitches and flaws of your electronic appliances with Geek Squad Online Support. Avail the best online support services for fixing your electronic appliance. Reach the experts at Geek Squad Online Support with 24/7 availability to avail best solutions for your troublesome devices. https://geek-customer-care.com/geek-squad-online-support.html

3 years, 9 months

1
0
0 0

Stuck removing osd with orch

by Ml Ml

Hello, yesterday i did: ceph osd purge 32 --yes-i-really-mean-it I also started to upgrade: ceph orch upgrade start --ceph-version 15.2.4 It seems its really gone: ceph osd crush remove osd.32 => device 'osd.32' does not appear in the crush map ceph orch ps: osd.32 ceph01 error 4h ago 2h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> root@ceph01:~# ceph orch osd rm status NAME HOST PGS STARTED_AT osd.32 ceph01 n/a 2020-07-29 09:50:11.026643 (some hours ago) root@ceph01:~# ceph health detail HEALTH_ERR Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2; Low space hindering backfill (add storage if this doesn't resolve itself): 105 pgs backfill_toofull; Degraded data redundancy: 9882/33167949 objects degraded (0.030%), 1 pg degraded, 1 pg undersized [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2 Module 'cephadm' has failed: auth get failed: failed to find osd.32 in keyring retval: -2 Any idea what i would try to remove that OSD? Thanks, Michael

3 years, 9 months

1
0
0 0

mon tried to load "000000.sst" which doesn't exist when recovering from osds

by Yu Wei

Hi, I deployed rook v0.8.3 with ceph 12.2.7. This is production system deployed for a long time. Because unknown reason, mon couldn't form quorum anymore and I tried to restore mon from osd by following document below, https://github.com/ceph/ceph/blob/v12.2.7/doc/rados/troubleshooting/trouble… After collecting cluster map data, replace store.db and restart mon, monitor log indicated that it tried to load "000000.sst" which was not existed. Log also indicated that mon found all .sst files during startup. Detailed log as below. 2020-07-28 09:44:38.100932 I | rook-ceph-mon2: 2020-07-28 09:44:38.100799 7f2e4abd0ec0 4 rocksdb: CURRENT file: CURRENT 2020-07-28 09:44:38.100946 I | rook-ceph-mon2: 2020-07-28 09:44:38.100951 I | rook-ceph-mon2: 2020-07-28 09:44:38.100847 7f2e4abd0ec0 4 rocksdb: IDENTITY file: IDENTITY 2020-07-28 09:44:38.100958 I | rook-ceph-mon2: 2020-07-28 09:44:38.100963 I | rook-ceph-mon2: 2020-07-28 09:44:38.100865 7f2e4abd0ec0 4 rocksdb: MANIFEST file: MANIFEST-000014 size: 284 Bytes 2020-07-28 09:44:38.100967 I | rook-ceph-mon2: 2020-07-28 09:44:38.100972 I | rook-ceph-mon2: 2020-07-28 09:44:38.100869 7f2e4abd0ec0 4 rocksdb: SST files in /var/lib/rook/rook-ceph-mon2/data/store.db dir, Total Num: 3, files: 000004.sst 000007.sst 000010.sst 2020-07-28 09:44:38.100976 I | rook-ceph-mon2: 2020-07-28 09:44:38.100981 I | rook-ceph-mon2: 2020-07-28 09:44:38.100872 7f2e4abd0ec0 4 rocksdb: Write Ahead Log file in /var/lib/rook/rook-ceph-mon2/data/store.db: 000015.log size: 0 ; 2020-07-28 09:44:38.100985 I | rook-ceph-mon2: 2020-07-28 09:44:38.100989 I | rook-ceph-mon2: 2020-07-28 09:44:38.100874 7f2e4abd0ec0 4 rocksdb: Options.error_if_exists: 0 ... 2020-07-28 09:44:38.101528 I | rook-ceph-mon2: 2020-07-28 09:44:38.101467 7f2e4abd0ec0 4 rocksdb: Fast CRC32 supported: 1 2020-07-28 09:44:38.104667 I | rook-ceph-mon2: 2020-07-28 09:44:38.104317 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000014 2020-07-28 09:44:38.104726 I | rook-ceph-mon2: 2020-07-28 09:44:38.104926 I | rook-ceph-mon2: 2020-07-28 09:44:38.104582 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]: ... 2020-07-28 09:44:38.105633 I | rook-ceph-mon2: 2020-07-28 09:44:38.104857 7f2e4abd0ec0 4 rocksdb: Options.report_bg_io_stats: 0 2020-07-28 09:44:38.111205 I | rook-ceph-mon2: 2020-07-28 09:44:38.110905 7f2e4abd0ec0 2 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:1062] Unable to load table properties for file 0 --- IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.111266 I | rook-ceph-mon2: 2020-07-28 09:44:38.111693 I | rook-ceph-mon2: 2020-07-28 09:44:38.110999 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:/var/lib/rook/rook-ceph-mon2/data/store.db/MANIFEST-000014 succeeded,manifest_file_number is 14, next_file_number is 17, last_sequence is 111, log_number is 0,prev_log_number is 0,max_column_family is 0 2020-07-28 09:44:38.111723 I | rook-ceph-mon2: 2020-07-28 09:44:38.111732 I | rook-ceph-mon2: 2020-07-28 09:44:38.111006 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 13 2020-07-28 09:44:38.111738 I | rook-ceph-mon2: 2020-07-28 09:44:38.111746 I | rook-ceph-mon2: 2020-07-28 09:44:38.111101 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work 2020-07-28 09:44:38.111764 I | rook-ceph-mon2: 2020-07-28 09:44:38.111214 7f2e4abd0ec0 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl.cc:343] Shutdown complete 2020-07-28 09:44:38.112077 I | rook-ceph-mon2: 2020-07-28 09:44:38.111862 7f2e4abd0ec0 -1 rocksdb: Corruption: Can't access /000000.sst: IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.112096 I | rook-ceph-mon2: 2020-07-28 09:44:38.112103 I | rook-ceph-mon2: 2020-07-28 09:44:38.111941 7f2e4abd0ec0 -1 error opening mon data directory at '/var/lib/rook/rook-ceph-mon2/data': (22) Invalid argument 2020-07-28 09:44:38.116813 I | rook-ceph-mon2: 2020-07-28 09:44:38.111862 7f2e4abd0ec0 -1 rocksdb: Corruption: Can't access /000000.sst: IO error: /var/lib/rook/rook-ceph-mon2/data/store.db/000000.sst: No such file or directory 2020-07-28 09:44:38.116874 I | rook-ceph-mon2: 2020-07-28 09:44:38.116883 I | rook-ceph-mon2: 2020-07-28 09:44:38.111941 7f2e4abd0ec0 -1 error opening mon data directory at '/var/lib/rook/rook-ceph-mon2/data': (22) Invalid argument failed to run mon. failed to start mon: Failed to complete 'rook-ceph-mon2': exit status 1. It's a little weird that it tried to load "000000.sst" although rook found the correct three .sst files during startup. There is no "000000.sst" related content in store.db. Any advice for this problem? Is it possible that I executed any wrong steps? Or is there any workaround for this? root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# ls -al store.db/ total 92 drwxr-xr-x 2 root root 188 Jul 28 01:35 . drwxr--r-- 3 root root 55 Jul 28 02:37 .. -rw-r--r-- 1 root root 56547 Jul 28 02:38 000004.sst -rw-r--r-- 1 root root 1179 Jul 28 02:38 000007.sst -rw-r--r-- 1 root root 1243 Jul 28 02:38 000010.sst -rw-r--r-- 1 root root 0 Jul 28 02:38 000015.log -rw-r--r-- 1 root root 16 Jul 28 02:38 CURRENT -rw-r--r-- 1 root root 37 Jul 28 02:38 IDENTITY -rw-r--r-- 1 root root 0 Jul 28 02:38 LOCK -rw-r--r-- 1 root root 284 Jul 28 02:38 MANIFEST-000014 -rw-r--r-- 1 root root 4620 Jul 28 02:38 OPTIONS-000014 -rw-r--r-- 1 root root 4620 Jul 28 02:38 OPTIONS-000017 root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# find . -type f | xargs grep "000000.sst" root@bagwig4:/var/lib/rook/rook-ceph-mon2/data# find . -type f | xargs grep "000000" ./store.db/OPTIONS-000017: delete_obsolete_files_period_micros=21600000000 ./store.db/OPTIONS-000017: memtable_prefix_bloom_size_ratio=0.000000 ./store.db/OPTIONS-000017: max_bytes_for_level_multiplier=10.000000 ./store.db/OPTIONS-000014: delete_obsolete_files_period_micros=21600000000 ./store.db/OPTIONS-000014: memtable_prefix_bloom_size_ratio=0.000000 ./store.db/OPTIONS-000014: max_bytes_for_level_multiplier=10.000000 Thanks, Jared, (韦煜） Software developer Interested in open source software, big data, Linux

3 years, 9 months

1
0
0 0

Does Cash app customer service advise me if an account gets checked?

by david william

At whatever point you are applying to confirm your character on the Cash app, at that point you will likewise get a warning of further advances. All things considered, you have to check your record, on the off chance that you need to go through more cash, or you need to utilize extra services on it. You just need to enter your complete name, date of birth, last four digits of SSN. In the event that there is an issue, contact Cash app customer service.https://www.cash-app-customer-service.com/

3 years, 9 months

1
0
0 0

Issue in getting a Cash App refund on account of headway tab? Call help gathering.

by david william

The advancement tab is one of the basic holds onto the item this is in all probability subject for the Cash App refund. As such, at the off peril which you have to get your coins back, with the guide of using then you may use the help this is given withinside the assistance targets or you may dial the wide variety to interface with the assist group with anticipating any help. https://www.contact-customer-service.net/blog/cash-app-refund/

3 years, 9 months

1
0
0 0

Current best practice for migrating from one EC profile to another?

by David Orman

Hi, As we expand our cluster (adding nodes), we'd like to take advantage of better EC profiles enabled by higher server/rack counts. I understand, as Ceph currently exists (15.2.4), there is no way to live-migrate from one EC profile to another on an existing pool, for example, from 4+2 to 17+3 when going from 7 nodes to 21. Is this correct? How are people accomplishing migrations such as this one (or 4+2 to 9+3, for example) with minimal disruption to services that utilize RBDs sitting on top of these pools? I found: https://ceph.io/geen-categorie/ceph-pool-migration/ , which requires effectively shutting down access during migration (which is doable, but not ideal) - and from what I've read - has some potential downsides (specifically referencing the cppool method). The pools are all EC for data, and an rbd pool exists for metadata. Thank you!

3 years, 9 months

3
2
0 0

July Ceph Science User Group Virtual Meeting

by Kevin Hrpcek

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday July 22nd. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20200722 We try to keep it to an hour or less. Ceph calendar event details: July 22, 2020 14:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

3 years, 9 months

2
1
0 0

repeatable crash in librbd1

by Johannes Naab

Hi, we observe crashes in librbd1 on specific workloads in virtual machines on Ubuntu 20.04 hosts with librbd1=15.2.4-1focal. The changes in https://github.com/ceph/ceph/commit/50694f790245ca90a3b8a644da7b128a7a148cc6 could be related, but do not easily apply against v15.2.4. We have collected several backtraces, using a reliable local reproducer. (This is a resent of the message stuck in the moderation queue since yesterday.) Best regards Johannes VM in libvirt with: <pre> <disk type='network' device='disk'> <driver name='qemu' type='raw' discard='unmap'/> <source protocol='rbd' name='pool/disk' index='4'>  </source> <iotune> <read_bytes_sec>209715200</read_bytes_sec> <write_bytes_sec>209715200</write_bytes_sec> <read_iops_sec>5000</read_iops_sec> <write_iops_sec>5000</write_iops_sec> <read_bytes_sec_max>314572800</read_bytes_sec_max> <write_bytes_sec_max>314572800</write_bytes_sec_max> <read_iops_sec_max>7500</read_iops_sec_max> <write_iops_sec_max>7500</write_iops_sec_max> <read_bytes_sec_max_length>60</read_bytes_sec_max_length> <write_bytes_sec_max_length>60</write_bytes_sec_max_length> <read_iops_sec_max_length>60</read_iops_sec_max_length> <write_iops_sec_max_length>60</write_iops_sec_max_length> </iotune> </disk> </pre> workload: <pre> fio --rw=write --name=test --size=10M timeout 30s fio --rw=write --name=test --size=20G timeout 3m fio --rw=write --name=test --size=20G --direct=1 timeout 1m fio --rw=randrw --name=test --size=20G --direct=1 timeout 10s fio --numjobs=8 --rw=randrw --name=test --size=1G --direct=1 # the backtraces are then observed while the following command is running fio --ioengine=libaio --iodepth=16 --numjobs=8 --rw=randrw --name=test --size=1G --direct=1 </pre> observed stack traces three times: <pre> #0 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x557f633e9400) at ./src/common/event_socket.h:32 #1 0x00007ffb9740ba34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:262 #2 0x00007ffb9740ce98 in librbd::io::AioCompletion::complete (this=0x557f633f7600) at ./src/librbd/io/AioCompletion.cc:104 #3 0x00007ffb9740d1a0 in librbd::io::AioCompletion::complete_request (this=0x557f633f7600, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #4 0x00007ffb9742fdca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x7ffb68364a60, r=4096) at ./src/librbd/io/ReadResult.cc:155 #5 0x00007ffb9728334d in Context::complete (this=0x7ffb68364a60, r=<optimized out>) at ./src/include/Context.h:77 #6 0x00007ffb9742c7d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x7ffb6832bed0, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #7 0x00007ffb9742c735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #8 0x00007ffb9755e942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x7ffb683394a0, r=r@entry=0) at ./src/include/Context.h:78 #9 0x00007ffb97562e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x7ffb683394a0, r=0) at ./src/log/SubsystemMap.h:72 #10 0x00007ffb970f4177 in librados::C_AioComplete::finish (this=0x7ffb68362e30, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #11 0x00007ffb970afd1d in Context::complete (this=0x7ffb68362e30, r=<optimized out>) at ./src/include/Context.h:77 #12 0x00007ffb8765791d in Finisher::finisher_thread_entry (this=0x557f63387a30) at ./src/common/Finisher.cc:66 #13 0x00007ffb9946f609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #14 0x00007ffb99396103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> once: <pre> #0 0x00007fe7a972169d in std::atomic<boost::lockfree::detail::tagged_ptr<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::load (__m=std::memory_order_acquire, this=<optimized out>) at /usr/include/c++/9/atomic:250 #1 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:311 #2 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #3 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x5613db630440) at ./src/librbd/io/AioCompletion.cc:276 #4 0x00007fe7a9721a34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:262 #5 0x00007fe7a9722e98 in librbd::io::AioCompletion::complete (this=0x5613dbbfa2c0) at ./src/librbd/io/AioCompletion.cc:104 #6 0x00007fe7a97231a0 in librbd::io::AioCompletion::complete_request (this=0x5613dbbfa2c0, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #7 0x00007fe7a9745dca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x5613dc6a5c90, r=4096) at ./src/librbd/io/ReadResult.cc:155 #8 0x00007fe7a959934d in Context::complete (this=0x5613dc6a5c90, r=<optimized out>) at ./src/include/Context.h:77 #9 0x00007fe7a97427d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x5613db8bae40, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #10 0x00007fe7a9742735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #11 0x00007fe7a9874942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x5613dbbfc900, r=r@entry=0) at ./src/include/Context.h:78 #12 0x00007fe7a9878e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x5613dbbfc900, r=0) at ./src/log/SubsystemMap.h:72 #13 0x00007fe7a940a177 in librados::C_AioComplete::finish (this=0x5613db85e6d0, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #14 0x00007fe7a93c5d1d in Context::complete (this=0x5613db85e6d0, r=<optimized out>) at ./src/include/Context.h:77 #15 0x00007fe79b65791d in Finisher::finisher_thread_entry (this=0x5613db369ee0) at ./src/common/Finisher.cc:66 #16 0x00007fe7ab788609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #17 0x00007fe7ab6af103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> twice: <pre> #0 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::extract_ptr (i=<error reading variable>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:113 #1 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::get_ptr (this=0xfffff40ee9c0) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:115 #2 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate_impl<false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:187 #3 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate<true, false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:168 #4 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::construct<true, false, librbd::io::AioCompletion*, boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node*> (arg2=<optimized out>, arg1=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:100 #5 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:302 #6 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #7 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x560f5e10c000) at ./src/librbd/io/AioCompletion.cc:276 #8 0x00007f8c8bcada34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x560f5edd5d30) at ./src/librbd/io/AioCompletion.cc:262 #9 0x00007f8c8bcaee98 in librbd::io::AioCompletion::complete (this=0x560f5edd5d30) at ./src/librbd/io/AioCompletion.cc:104 #10 0x00007f8c8bcaf1a0 in librbd::io::AioCompletion::complete_request (this=0x560f5edd5d30, r=r@entry=4096) at ./src/librbd/io/AioCompletion.cc:229 #11 0x00007f8c8bcd1dca in librbd::io::ReadResult::C_ObjectReadRequest::finish (this=0x7f8bc4342280, r=4096) at ./src/librbd/io/ReadResult.cc:155 #12 0x00007f8c8bb2534d in Context::complete (this=0x7f8bc4342280, r=<optimized out>) at ./src/include/Context.h:77 #13 0x00007f8c8bcce7d9 in librbd::io::ObjectDispatchSpec::C_Dispatcher::finish (this=0x7f8bc4326820, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:32 #14 0x00007f8c8bcce735 in librbd::io::ObjectDispatchSpec::C_Dispatcher::complete (this=<optimized out>, r=<optimized out>) at ./src/librbd/io/ObjectDispatchSpec.cc:23 #15 0x00007f8c8be00942 in librbd::io::ObjectRequest<librbd::ImageCtx>::finish (this=this@entry=0x7f8bc42b2a30, r=r@entry=0) at ./src/include/Context.h:78 #16 0x00007f8c8be04e3b in librbd::io::ObjectReadRequest<librbd::ImageCtx>::handle_read_object (this=0x7f8bc42b2a30, r=0) at ./src/log/SubsystemMap.h:72 #17 0x00007f8c8b996177 in librados::C_AioComplete::finish (this=0x7f8bc43423b0, r=<optimized out>) at ./src/librados/AioCompletionImpl.h:140 #18 0x00007f8c8b951d1d in Context::complete (this=0x7f8bc43423b0, r=<optimized out>) at ./src/include/Context.h:77 #19 0x00007f8c82f4291d in Finisher::finisher_thread_entry (this=0x560f5f44b180) at ./src/common/Finisher.cc:66 #20 0x00007f8c91e1a609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #21 0x00007f8c91d41103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre> once: <pre> #0 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::extract_ptr (i=<error reading variable>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:113 #1 boost::lockfree::detail::tagged_ptr<boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::freelist_node>::get_ptr (this=0xfffff40ee9c0) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/tagged_ptr_ptrcompression.hpp:115 #2 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate_impl<false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:187 #3 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::allocate<true, false> (this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:168 #4 boost::lockfree::detail::freelist_stack<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node, std::allocator<boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node> >::construct<true, false, librbd::io::AioCompletion*, boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::node*> (arg2=<optimized out>, arg1=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/detail/freelist.hpp:100 #5 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::do_push<false> (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:302 #6 boost::lockfree::queue<librbd::io::AioCompletion*, boost::lockfree::allocator<std::allocator<void> > >::push (t=<optimized out>, this=<optimized out>) at ./obj-x86_64-linux-gnu/boost/include/boost/lockfree/queue.hpp:280 #7 librbd::io::AioCompletion::complete_event_socket (this=this@entry=0x55d1b89113c0) at ./src/librbd/io/AioCompletion.cc:276 #8 0x00007f275f6d7a34 in librbd::io::AioCompletion::complete_external_callback (this=this@entry=0x55d1b8a7a0d0) at ./src/librbd/io/AioCompletion.cc:262 #9 0x00007f275f6d8e98 in librbd::io::AioCompletion::complete (this=0x55d1b8a7a0d0) at ./src/librbd/io/AioCompletion.cc:104 #10 0x00007f275f6d91a0 in librbd::io::AioCompletion::complete_request (this=0x55d1b8a7a0d0, r=<optimized out>) at ./src/librbd/io/AioCompletion.cc:229 #11 0x00007f275f54f34d in Context::complete (this=0x7f268833f850, r=<optimized out>) at ./src/include/Context.h:77 #12 0x00007f274f6b647b in ThreadPool::worker (this=0x55d1b9d04860, wt=<optimized out>) at ./src/common/WorkQueue.cc:118 #13 0x00007f274f6b7545 in ThreadPool::WorkThread::entry (this=<optimized out>) at ./src/common/WorkQueue.h:466 #14 0x00007f276173e609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #15 0x00007f2761665103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 </pre>

3 years, 9 months

2
6
0 0

slow ops on one osd makes all my buckets unavailable

by Zhenshi Zhou

Hi, My harbor registry uses ceph object storage to save the images. But I couldn't pull/push images from harbor a few moments ago. Ceph was in warning health status in the same time. The cluster just had a warning message said that osd.24 has slow ops. I check the ceph-osd.24.log, and showed as below: *2020-07-28 19:01:40.599 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:41.558 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:42.579 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:43.566 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:44.588 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:45.627 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:46.674 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144787 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[] ondisk+write+known_if_redirected e4324)2020-07-28 19:01:47.701 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:48.729 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:49.729 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)2020-07-28 19:01:50.889 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.166289.0:34144852 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)......* *......* *2020-07-28 21:03:35.053 7f907a39c700 -1 osd.24 4324 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.166298.0:34904067 17.4f 17:f29d8b20:::.dir.313c8244-fe4d-4d46-bf9b-0e33e46be041.157033.2:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e4324)* After restarted osd.24, the cluster became health again, so was harbor. What confuse me is that why my harbor couldn't get data from its bucket while the log indicated that there was a client block on the other bucket. I don't think some slow ops on an osd has any bad effect on all buckets. Any idea is appreciated. Thanks

3 years, 9 months

1
0
0 0

S3 bucket lifecycle not deleting old objects

by Alex Hussein-Kershaw

Hello, I have a problem that old versions of S3 objects are not being deleted. Can anyone advise as to why? I'm using Ceph 14.2.9. I expect old versions of S3 objects to be deleted after 3 days as per my lifecycle config on the bucket: { "Rules": [ { "Status": "Enabled", "Prefix": "", "NoncurrentVersionExpiration": { "NoncurrentDays": 3 }, "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "S3 scsdata bucket: Tidy up old versions" } ] } I have an object with 3 versions below (and is much older than 3 days): [root@hera hera_sdc] /usr/bin> aws s3api --endpoint http://127.3.3.3:7480 list-object-versions --bucket hera-scsdata --key 84/46/2020060508501821902143658709-Subscriber | grep -B 4 -A 6 84/46/2020060508501821902143658709-Subscriber "LastModified": "2020-06-05T08:58:19.644Z", "VersionId": "FUdZIehBu3sgRbNJSmZwj3VHWs1ednH", "ETag": "\"a18286c50a7323efe58497eb97d6dc9d\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": true, "Size": 4440 -- "LastModified": "2020-06-05T08:58:17.943Z", "VersionId": "JVKGMJQS-l7xKQuqdfn4QsEY5WLEosj", "ETag": "\"87e9953af436b702afb80d457f1d73cb\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": false, "Size": 4408 -- "LastModified": "2020-06-05T08:50:19.167Z", "VersionId": "-RSNSCDvGj83f4DZ11s8YZ2KaxT8T.a", "ETag": "\"a68ec68ce825e009ee9a70cfdae9c794\"", "StorageClass": "STANDARD", "Key": "84/46/2020060508501821902143658709-Subscriber", "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, "IsLatest": false, "Size": 4256 -- ], "NextKeyMarker": "85/49/20200604163626B4C712312312302641-Subscriber", "MaxKeys": 1000, "Prefix": "", "KeyMarker": "84/46/2020060508501821902143658709-Subscriber", "DeleteMarkers": [ { "Owner": { "DisplayName": "hera EAS S3 user", "ID": "hera" }, So those objects still being present seems to be in conflict with the config I have set? Thanks, Alex

3 years, 9 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users