- ceph-users - lists.ceph.io

Google Summer of Code 2020
by Alastair Dewhurst - UKRI STFC 23 Jan '20

23 Jan '20

Hello I apologies if I have missed an announcement, but I can’t find anything regarding if Ceph will participate in the GSoC this year? The only thing I could find was last years announcement (see below) and webpage. My organisation has some ideas for Ceph projects, however we could submit them under a different banner organisation if necessary. Thanks Alastair On 17 Jan 2019, at 08:12, Mike Perez <miperez(a)redhat.com<mailto:miperez@redhat.com>> wrote: Hello everyone, We're getting ready for the next round of Google Summer of Code and Outreachy. Ali Maredia and I will help organize the Ceph project for each program. We are looking to have project ideas for Google Summer of Code by February 4th as our project application deadline is due by February 6th. https://ceph.com/contribute/gsoc2019/ https://summerofcode.withgoogle.com/how-it-works/ Outreachy applicants in mid-February through the end of March will begin selecting open source projects to work on and make small contributions. On April 12th we hope to have some projects set for Outreachy so we can start selecting applicants. End of May will begin the internships. https://www.outreachy.org/mentor/ You can submit project ideas for both programs on this etherpad. https://pad.ceph.com/p/project-ideas Stay tuned for more updates. -- Mike Perez (thingee) _______________________________________________ ceph-users mailing list ceph-users(a)lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

cephfs kernel mount option uid?
by Marc Roos 23 Jan '20

23 Jan '20

Is it possible to mount a cephfs with a specific uid or gid? To make it available to a 'non-root' user?

2 1

Problem : "1 pools have many more objects per pg than average"
by St-Germain, Sylvain (SSC/SPC) 23 Jan '20

23 Jan '20

///////////////////////// Problem /////////////////////////// I've got a Warning on my cluster that I cannot remove : "1 pools have many more objects per pg than average" Does somebody has some insight ? I think it's normal to have this warning because I have just one pool in use, but how can I remove this warning ? Thx ! ///////////////////////// INFORMATION /////////////////////////// *** Here's some information about the cluster # ceph health detail HEALTH_WARN 1 pools have many more objects per pg than average MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average pool default.rgw.buckets.data objects per pg (50567) is more than 14.1249 times cluster average (3580) # sudo ceph-conf -D | grep mon_pg_warn_max_object_skew mon_pg_warn_max_object_skew = 10.000000 # ceph daemon mon.dao-wkr-04 config show | grep mon_pg_warn_max_object_skew "mon_pg_warn_max_object_skew": "10.000000" # ceph -v ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 873 TiB 823 TiB 50 TiB 51 TiB 5.80 TOTAL 873 TiB 823 TiB 50 TiB 51 TiB 5.80 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL .rgw.root 11 3.5 KiB 8 1.5 MiB 0 249 TiB default.rgw.control 12 0 B 8 0 B 0 249 TiB default.rgw.meta 13 52 KiB 186 34 MiB 0 249 TiB default.rgw.log 14 0 B 207 0 B 0 249 TiB default.rgw.buckets.index 15 1.2 GiB 131 1.2 GiB 0 249 TiB cephfs_data 29 915 MiB 202 1.5 GiB 0 467 TiB cephfs_metadata 30 145 KiB 23 2.1 MiB 0 249 TiB default.rgw.buckets.data 31 30 TiB 12.95M 50 TiB 6.32 467 TiB # ceph osd dump | grep default.rgw.buckets.data pool 31 'default.rgw.buckets.data' erasure size 8 min_size 6 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 9502 lfor 0/2191/7577 flags hashpspool stripe_width 20480 target_size_ratio 0.4 application rgw /////////////////////////// SOLUTION TRIED /////////////////////////// 1- I try to increase the value of mon_pg_warn_max_object_skew parameter #sudo ceph tell mon.* injectargs '--mon_pg_warn_max_object_skew 20' #sudo ceph tell osd.* injectargs '--mon_pg_warn_max_object_skew 20' + And reboot the monitor The parameter didn't change.

1 0

Fwd: OSD crash after change of osd_memory_target
by Martin Mlynář 23 Jan '20

23 Jan '20

Hi Igor, unfortunately same result: # ceph config dump WHO MASK LEVEL OPTION VALUE RO osd basic osd_memory_target 2147483648 # /usr/bin/ceph-osd -d --cluster ceph --id 0 --setuser ceph --setgroup ceph .... 0> 2020-01-23 10:48:04.436 7fc61b5b5c80 -1 *** Caught signal (Aborted) ** in thread 7fc61b5b5c80 thread_name:ceph-osd ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) 1: (()+0x12730) [0x7fc61c05d730] 2: (gsignal()+0x10b) [0x7fc61bb417bb] 3: (abort()+0x121) [0x7fc61bb2c535] 4: (()+0x8c983) [0x7fc61bef4983] 5: (()+0x928c6) [0x7fc61befa8c6] 6: (()+0x92901) [0x7fc61befa901] 7: (()+0x92b34) [0x7fc61befab34] 8: (()+0x5a3f53) [0x55ecdabb2f53] 9: (Option::size_t const md_config_t::get_val<Option::size_t>(ConfigValues const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const+0x81) [0x55ecdabb8c91] 10: (BlueStore::_set_cache_sizes()+0x15a) [0x55ecdb033d8a] 11: (BlueStore::_open_bdev(bool)+0x173) [0x55ecdb036b23] 12: (BlueStore::get_devices(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)+0xef) [0x55ecdb09d7ef] 13: (BlueStore::get_numa_node(int*, std::set<int, std::less<int>, std::allocator<int> >*, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)+0x7b) [0x55ecdb04571b] 14: (main()+0x2870) [0x55ecdab80440] 15: (__libc_start_main()+0xeb) [0x7fc61bb2e09b] 16: (_start()+0x2a) [0x55ecdabb2c6a] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. .... Best Regards, Martin Dne 22. 01. 20 v 16:33 Igor Fedotov napsal(a): > > Hi Martin, > > looks like a bug to me. > > You might want to remove all custom settings from config database and > try to set osd-memory-target only. > > Would it help? > > > Thanks, > > Igor > > On 1/22/2020 3:43 PM, Martin Mlynář wrote: >> >> >> Dne 21. 01. 20 v 21:12 Stefan Kooman napsal(a): >>> Quoting Martin Mlynář (nexus+ceph(a)smoula.net) >>> >>>> Do you think this could help? OSD does not even start, I'm getting a little >>>> lost how flushing caches could help. >>> I might have mis-understood. I though the OSDs crashed when you set the >>> config setting. >>> >>>> According to trace I suspect something around processing config values. >>> I've just set the same config setting on a test cluster and restarted an >>> OSD without problem. So, not sure what is going on there. >>> >>> Gr. Stefan >> >> I've compiled ceph-osd with debug symbols and got better backtrace: >> >> -24> 2020-01-22 13:12:53.614 7f83ed064700 4 set_mon_vals no >> callback set >> -23> 2020-01-22 13:12:53.614 7f83ee867700 10 monclient: discarding >> stray monitor message auth_reply(proto 2 0 (0) Success) v1 >> -22> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_crush_update_on_start = true >> -21> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_max_backfills = 64 >> -20> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_memory_target = 2147483648 >> -19> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_max_active = 40 >> -18> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_max_single_start = 1000 >> -17> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_sleep_hdd = 0.000000 >> -16> 2020-01-22 13:12:53.614 7f83ed064700 10 set_mon_vals >> osd_recovery_sleep_hybrid = 0.000000 >> -15> 2020-01-22 13:12:53.627 7f83f0276c40 0 set uid:gid to >> 64045:64045 (ceph:ceph) >> -14> 2020-01-22 13:12:53.627 7f83f0276c40 0 ceph version 14.2.6 >> (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable), process >> ceph-osd, pid 1111622 >> -13> 2020-01-22 13:12:53.649 7f83f0276c40 0 pidfile_write: ignore >> empty --pid-file >> -12> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> init /var/run/ceph/ceph-osd.6.asok >> -11> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> bind_and_listen /var/run/ceph/ceph-osd.6.asok >> -10> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command 0 hook 0x558051872fc0 >> -9> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command version hook 0x558051872fc0 >> -8> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command git_version hook 0x558051872fc0 >> -7> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command help hook 0x558051874220 >> -6> 2020-01-22 13:12:53.657 7f83f0276c40 5 asok(0x5580518fa000) >> register_command get_command_descriptions hook 0x558051874260 >> -5> 2020-01-22 13:12:53.657 7f83ed865700 5 asok(0x5580518fa000) >> entry start >> -4> 2020-01-22 13:12:53.670 7f83f0276c40 5 object store type is >> bluestore >> -3> 2020-01-22 13:12:53.675 7f83f0276c40 1 bdev create path >> /var/lib/ceph/osd/ceph-6/block type kernel >> -2> 2020-01-22 13:12:53.675 7f83f0276c40 1 bdev(0x5580518f3f80 >> /var/lib/ceph/osd/ceph-6/block) open path /var/lib/ceph/osd/ceph-6/block >> -1> 2020-01-22 13:12:53.675 7f83f0276c40 1 bdev(0x5580518f3f80 >> /var/lib/ceph/osd/ceph-6/block) open size 3000588304384 >> (0x2baa1000000, 2.7 TiB) block_size 4096 (4 KiB) rotational discard >> not supported >> 0> 2020-01-22 13:12:53.714 7f83f0276c40 -1 *** Caught signal >> (Aborted) ** >> in thread 7f83f0276c40 thread_name:ceph-osd >> >> ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) >> nautilus (stable) >> 1: (()+0x2c19654) [0x558045ec6654] >> 2: (()+0x12730) [0x7f83f0d1f730] >> 3: (gsignal()+0x10b) [0x7f83f08027bb] >> 4: (abort()+0x121) [0x7f83f07ed535] >> 5: (()+0x8c983) [0x7f83f0bb5983] >> 6: (()+0x928c6) [0x7f83f0bbb8c6] >> 7: (()+0x92901) [0x7f83f0bbb901] >> 8: (()+0x92b34) [0x7f83f0bbbb34] >> * 9: (void boost::throw_exception<boost::bad_get>(boost::bad_get >> const&)+0x7b) [0x5580454d5430]* >> * 10: (Option::size_t&& boost::relaxed_get<Option::size_t, >> boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>(boost::variant<boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>&&)+0x5b) [0x5580454d6223]* >> 11: (Option::size_t&& boost::strict_get<Option::size_t, >> boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>(boost::variant<boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>&&)+0x20) [0x5580454d4a39] >> 12: (Option::size_t&& boost::get<Option::size_t, boost::blank, >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >, unsigned long, long, double, bool, >> entity_addr_t, entity_addrvec_t, std::chrono::duration<long, >> std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>(boost::variant<boost::blank, std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, unsigned long, long, >> double, bool, entity_addr_t, entity_addrvec_t, >> std::chrono::duration<long, std::ratio<1l, 1l> >, Option::size_t, >> uuid_d>&&)+0x20) [0x5580454d1ed7] >> 13: (Option::size_t const >> md_config_t::get_val<Option::size_t>(ConfigValues const&, >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > const&) const+0x48) [0x5580454ce882] >> 14: (Option::size_t const >> ConfigProxy::get_val<Option::size_t>(std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > const&) const+0x58) >> [0x5580454cb9b8] >> 15: (BlueStore::_set_cache_sizes()+0x159) [0x558045ce2213] >> 16: (BlueStore::_open_bdev(bool)+0x301) [0x558045ce6be3] >> 17: >> (BlueStore::get_devices(std::set<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, >> std::less<std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > >, >> std::allocator<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > > >*)+0xf9) >> [0x558045d0f16d] >> 18: (BlueStore::get_numa_node(int*, std::set<int, std::less<int>, >> std::allocator<int> >*, std::set<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> >, >> std::less<std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > >, >> std::allocator<std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > > >*)+0x79) >> [0x558045d0eb55] >> 19: (main()+0x3aae) [0x5580454c2460] >> 20: (__libc_start_main()+0xeb) [0x7f83f07ef09b] >> 21: (_start()+0x2a) [0x5580454bda2a] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> needed to interpret this. >> >> And managed some gdb debugging (int BlueStore::_set_cache_sizes()): >> >> (gdb) n >> 4116 cache_autotune_interval = >> (gdb) n >> 4117 >> cct->_conf.get_val<double>("bluestore_cache_autotune_interval"); >> (gdb) p cache_autotune_interval >> $3 = 5 >> (gdb) n >> 4118 osd_memory_target = >> cct->_conf.get_val<Option::size_t>("osd_memory_target"); >> (gdb) s >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> >::basic_string<std::allocator<char> > >> (this=0x7fffffffc140, __s=0x555558d26c2f "osd_memory_target", __a=...) >> at /usr/include/c++/8/bits/basic_string.h:515 >> 515 : _M_dataplus(_M_local_data(), __a) >> (gdb) n >> 516 { _M_construct(__s, __s ? __s + traits_type::length(__s) >> : __s+npos); } >> (gdb) >> terminate called after throwing an instance of >> 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_get> >> >' >> what(): boost::bad_get: failed value get using boost::get >> >> But there I'm stuck. GDBing c++ code is a really dark sorcery for me. >> >> Other get_vals look fine, maybe get_val<*Option::size_t*> is the >> problem? It looks like trouble outside of ceph - what system are you >> testing on? This is debian with official debian build from >> buster-backports. Maybe some of debian's patches? >> >> -- >> Martin Mlynář >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users(a)lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

1 0

Ceph at DevConf and FOSDEM
by Mike Perez 23 Jan '20

23 Jan '20

Hi Cephers, We will have a booth at DevConf this year! https://www.devconf.info/cz/ If you're interested in helping at the booth at DevConf, please reply to me directly, and I can provide more details. I'm still working on getting a shared booth for FOSDEM with the CentOS table. Also, I will try to get us signed with a BoF session. If you're interested, please also let me know directly. Thanks! https://fosdem.org/2020/ -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 1

Upcoming Ceph Days for 2020
by Mike Perez 23 Jan '20

23 Jan '20

Hi Cephers, We have just posted some upcoming Ceph Days. We are looking for sponsors and content: * Ceph Day Istanbul: March 17 * Ceph Day Oslo: May 13 * Ceph Day Vancouver: May 13 https://ceph.com/cephdays/ Also don't forget about our big event Cephalocon Seoul March 3-5. Registration, schedule. sponsorship and hotel information is available: https://ceph.io/cephalocon/seoul-2020/ If you have any questions about these events, please contact us at events(a)ceph.io. Any help with promoting these events is greatly appreciated. Thanks! -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Migrate Jewel from leveldb to rocksdb
by Robert LeBlanc 22 Jan '20

22 Jan '20

In the last release of Jewel [0] it mentions that omap data can be stored in rocksdb instead of leveldb. We are seeing high latencies from compaction of leveldb on our Jewel cluster (can't upgrade at this time). I installed the latest version, but apparently that is not enough to do the conversion. Is there a way to move OSDs from leveldb to rocksdb without trashing and rebuilding all the osds? [0] https://docs.ceph.com/docs/master/releases/jewel/ Thanks, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

3 3

Auto create rbd snapshots
by Marc Roos 22 Jan '20

22 Jan '20

Is it possible to schedule the creation of snapshots on specific rbd images within ceph?

1 0

Large omap objects in radosgw .usage pool: is there a way to reshard the rgw usage log?
by Ingo Reimann 22 Jan '20

22 Jan '20

Hi All >On 09/10/2019 09:07, Florian Haas wrote: >[...] >the question with about resharding the usage log still stands. (The untrimmed usage log, in my case, would have blasted the old 2M keys threshold, too.) > >Cheers, Florian Is there any new wisdom about resharding the usage log for one user? Since Nautilus we get a HEALTH_WARN after 3 weeks of the month because the usage data of one single user reaches the threshold for large omap warnings - which I already increased to 1M. At start of month, we truncate the usage data so we are save again for a while. Cheers, ingo -- ngo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

1 0

Cephalocon early-bird registration ends today
by Sage Weil 21 Jan '20

21 Jan '20

Hi everyone, Quick reminder that the early-bird registration for Cephalocon Seoul (Mar 3-5) ends tonight! We also have the hotel booking link and code up on the site (finally--sorry for the delay). https://ceph.io/cephalocon/seoul-2020/ Hope to see you there! sage

1 0