- ceph-users - lists.ceph.io

ceph and openstack throttling experience
by Marcel Kuiper 10 Jun '21

10 Jun '21

Hi We're running ceph nautilus 14.2.21 (going to octopus latest in a few weeks) as volume and instance backend for our openstack vm's. Our clusters run somewhere between 500 - 1000 OSDs on SAS HDDs with NVMe's as journal and db device Currently we do not have our vm's capped on iops and throughput. We regularly get slowops warnings (once or twice per day) and wonder whether there are more users with sort of the same setup that do throttle their openstack vm's. - What kind of numbers are used in the field for IOPS and throughput limiting? - As a side question, is there an easy way to get rid of the slowops warning besides restarting the involved osd. Otherwise the warning seems to stay forever Regards Marcel

2 3

Ceph Octopus - How to customize the Grafana configuration
by Ralph Soika 10 Jun '21

10 Jun '21

Hello, I have installed and bootsraped a Ceph manager node via cephadm and the options: --initial-dashboard-user admin --initial-dashboard-password [PASSWORD] --dashboard-password-noupdate Everything works fine. I also have the Grafana Board to monitor my cluster. But the access to Grafana is open for anonymous users because of the grafana.ini template with the option: [auth.anonymous] enabled = true I can't figure out how to tweak the default grafana.ini file. Can someone help me how to do this? I tried to do this with the command: # ceph config-key set mgr/cephadm/services/grafana/grafana.ini \ -i /tmp//grafana.ini.j2 # ceph orch reconfig grafana But without any effect. I also did not really understand where I should place the grafana.ini file on my Host? Thanks for any help === Ralph --

2 3

nautilus: rbd ls returns ENOENT for some images
by Peter Lieven 10 Jun '21

10 Jun '21

Hi, we currently run into an issue where a rbd ls for a namespace returns ENOENT for some of the images in that namespace. /usr/bin/rbd --conf=XXX --id XXX ls 'mypool/28ef9470-76eb-4f77-bc1b-99077764ff7c' -l --format=json 2021-06-09 11:03:34.916 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55cacccc2390 fail: (2) No such file or directory 2021-06-09 11:03:34.916 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55caccd2b920 fail: (2) No such file or directory 2021-06-09 11:03:34.920 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55caccd9b4e0 fail: (2) No such file or directory rbd: error opening 34810ac2-3112-4fef-938c-b76338b0eeaf.raw: (2) No such file or directory rbd: error opening c9882583-6dd5-4eca-bb82-3e81f7d63fa9.raw: (2) No such file or directory rbd: error opening 5d5251d1-f017-4382-845c-65e504683742.raw: (2) No such file or directory 2021-06-09 11:03:34.924 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55cacce07b00 fail: (2) No such file or directory rbd: error opening c625b898-ec34-4446-9455-d2b70d9e378f.raw: (2) No such file or directory 2021-06-09 11:03:34.924 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55caccd7cce0 fail: (2) No such file or directory rbd: error opening 990c4bbe-6a7b-4adf-aab8-432e18d79e58.raw: (2) No such file or directory 2021-06-09 11:03:34.924 7f2225ffb700 -1 librbd::io::AioCompletion: 0x55cacce336f0 fail: (2) No such file or directory rbd: error opening 7382eb5b-a3eb-41e2-89b6-512f7b1d86c0.raw: (2) No such file or directory [{"image":"108600c6-2312-4d61-9f5b-35b351112512.raw","size":31457280000,"format":2,"lock_type":"exclusive"},{"image":"1292ef0c-2333-44f1-be30-39105f7d176e.raw","size":262149242880,"format":2,"lock_type":"exclusive"},{"image":"8cda5c3f-cdbd-42f4-918f-1480354e7965.raw","size":262149242880,"format":2,"lock_type":"exclusive"}] rbd: listing images failed: (2) No such file or directory The way to trigger this state was that the images which show "No such file or directory" were deleted with rbd rm, but the operation was interrupted (rbd process was killed) due to a timeout. What is the best way to recover from this and how to properly clean up? Release is nautilus 14.2.20 Thanks, Peter

2 4

Integration of openstack to ceph
by Michel Niyoyita 10 Jun '21

10 Jun '21

Dear Ceph Users, Anyone can help on the guidance of how I can integrate ceph to openstack ? especially RGW. Regards Michel

3 2

delete stray OSD daemon after replacing disk
by mabi 10 Jun '21

10 Jun '21

Hello, I replaced an OSD disk on one of my Nautilus OSD node which created a new osd number. Now ceph shows that there is one cephadm stray daemon (the old OSD #1 which I replaced) and which I can't remove as you can see below: # ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.1 on host ceph1e not managed by cephadm # ceph orch daemon rm osd.1 --force Error EINVAL: Unable to find daemon(s) ['osd.1'] Is there another command I am missing? Best regards, Mabi

3 4

1 daemons have recently crashed
by feng.zhang＠orcadt.com 10 Jun '21

10 Jun '21

Hi； ceph -s HEALTH_WARN [root@node-1 ~]# ceph crash info 2021-06-07T10:40:07.248249Z_c98563f9-78ef-4b9f-9f6b-816d2d2a333d { "backtrace": [ "(()+0xf5f0) [0x7fbb55b8f5f0]", "(std::_Rb_tree<boost::intrusive_ptr<AsyncConnection>, boost::intrusive_ptr<AsyncConnection>, std::_Identity<boost::intrusive_ptr<AsyncConnection> >, std::less<boost::intrusive_ptr<AsyncConnection> >, std::allocator<boost::intrusive_ptr<AsyncConnection> > >::find(boost::intrusive_ptr<AsyncConnection> const&) const+0x24) [0x556347e234b4]", "(AsyncConnection::_stop()+0xa7) [0x556347e1da87]", "(ProtocolV2::stop()+0x88) [0x556347e44e78]", "(ProtocolV2::_fault()+0x65) [0x556347e44fe5]", "(ProtocolV2::handle_read_frame_preamble_main(std::unique_ptr<ceph::buffer::v15_2_0::ptr_node, ceph::buffer::v15_2_0::ptr_node::disposer>&&, int)+0x30a) [0x556347e5ad5a]", "(ProtocolV2::run_continuation(Ct<ProtocolV2>&)+0x34) [0x556347e45b44]", "(AsyncConnection::process()+0x794) [0x556347e208e4]", "(EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa25) [0x556347c819d5]", "(()+0xd3c3e7) [0x556347c873e7]", "(()+0x1224c5f) [0x55634816fc5f]", "(()+0x7e65) [0x7fbb55b87e65]", "(clone()+0x6d) [0x7fbb54a4b88d]" ], "ceph_version": "2.0.11", "crash_id": "2021-06-07T10:40:07.248249Z_c98563f9-78ef-4b9f-9f6b-816d2d2a333d", "entity_name": "osd.45", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "7 (Core)", "os_version_id": "7", "process_name": "ceph-osd", "stack_sig": "897fe7f6bf2184fafd5b8a29905a147cb66850db318f6e874292a278aeb615bb", "timestamp": "2021-06-07T10:40:07.248249Z", "utsname_hostname": "node-8", "utsname_machine": "x86_64", "utsname_release": "5.4.22-1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Mon Mar 30 13:15:02 EDT 2020" } [root@node-1 ~]# Best Regards！ -------------------------------------- zhangfeng --------------------------------------- 西安奥卡云数据科技有限公司 ORCA DATA TECHNOLOGY (XI'AN) CO., LTD Email:feng.zhang@orcadt.com Phone: 13992881878 website: https://www.orcadt.com

1 0

Performance (RBD) regression after upgrading beyond v15.2.8
by Wido den Hollander 09 Jun '21

09 Jun '21

Hi, While doing some benchmarks I have two identical Ceph clusters: 3x SuperMicro 1U AMD Epyc 7302P 16C 256GB DDR 4x Samsung PM983 1,92TB 100Gbit networking I tested on such a setup with v16.2.4 with fio: bs=4k qd=1 IOps: 695 That was very low as I was expecting at least >1000 IOps. I checked with the second Ceph cluster which was still running v15.2.8, the result: 1364 IOps. I then upgraded from 15.2.8 to 15.2.13: 725 IOps Looking at the differences between v15.2.8 and v15.2.8 of options.cc I saw these options: bluefs_buffered_io: false -> true bluestore_cache_trim_max_skip_pinned: 1000 -> 64 The main difference seems to be 'bluefs_buffered_io', but in both cases this was already explicitly set to 'true'. So anything beyond 15.2.8 is right now giving me a much lower I/O performance with Queue Depth = 1 and Block Size = 4k. 15.2.8: 1364 IOps 15.2.13: 725 IOps 16.2.4: 695 IOps Has anybody else seen this as well? I'm trying to figure out where this is going wrong. Wido

3 3

Is it safe to mix Octopus and Pacific mons?
by Vladimir Brik 09 Jun '21

09 Jun '21

Hello My attempt to upgrade from Octopus to Pacific ran into issues, and I currently have one 16.2.4 mon and two 15.2.12 mons. Is this safe to run the cluster like this or should I shut down the 16.2.4 mon until I figure out what to do next with the upgrade? Thanks, Vlad

2 1

Upgrade to 16 failed: wrong /sys/fs/cgroup path
by Vladimir Brik 09 Jun '21

09 Jun '21

Hello My upgrade from 15.2.12 to 16.2.4 is stuck because a mon daemon failed to upgrade. Systemctl status of the mon showed this error: Error: open /sys/fs/cgroup/cpuacct,cpu/system.slice/... It turns out there is no /sys/fs/cgroup/cpuacct,cpu directory on my system. Instead, I have /sys/fs/cgroup/cpu,cpuacct. Symlinking them appears to have solved the immediate problem, but if I proceed with the upgrade to 16.2.4, after reboot, all ceph daemons will probably fail to start. Is this an issue with ceph, podman (2.1.1), or the fact that I am running Centos7? Is it possible to upgrade from 15.2.12 to 16.2.4 on Centos7? I thought that installing a version of podman that is compatible with both would suffice, but apparently not... Vlad

1 0

omap sizes
by Szabo, Istvan (Agoda) 09 Jun '21

09 Jun '21

Hi, Is there a way to check the omap sizes in the index pool? Either key numbers and size also? This one doesn't work: https://ceph.com/geen-categorie/get-omap-keyvalue-size/ Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0