- ceph-users - lists.ceph.io

high number of kernel clients per osd slow down
by Andrej Filipcic 19 Mar '21

19 Mar '21

Hi, I am testing 15.2.10 on a large cluster (RH8). cephfs pool (size=1) with 122 nvme OSDs works fine till the number of clients is relatively low. Writing from 400 kernel clients (ior benchmark), 8 streams each, causes issues. Writes are initially fast at 100GB/s but then they drop to <1GB/s after few minutes, while the OSDs use 300% cpu each. My guess is that the OSDs are overloaded with requests from too many clients, since it does not happen till there are ~3-4 streams OSD. The OSDs log do not show anything problematic. tried to increase osd_op_num_threads_per_shard_ssd, did not help. Restarting OSDs recovers the situation for few minutes. Writing to HDD pool with 1500 HDDs does not have any issues at all under same conditions. Any hints, settings to improve this? Cheers, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

2 2

March Ceph Science Virtual User Group Meeting
by Kevin Hrpcek 19 Mar '21

19 Mar '21

Hey all, We will be having a Ceph science/research/big cluster call on Wednesday March 24th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research environments where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. https://pad.ceph.com/p/Ceph_Science_User_Group_20210324 <https://pad.ceph.com/p/Ceph_Science_User_Group_20200923> We try to keep it to an hour or less. Ceph calendar event details: March 24th, 2021 15:00 UTC 4pm Central European 10am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://bluejeans.com/908675367?src=calendarLink To join from a Red Hat Deskphone or Softphone, dial: 84336. Connecting directly from a room system? 1.) Dial: 199.48.152.152 or bjn.vc 2.) Enter Meeting ID: 908675367 Just want to dial in on your phone? 1.) Dial one of the following numbers: 408-915-6466 (US) See all numbers: https://www.redhat.com/en/conference-numbers 2.) Enter Meeting ID: 908675367 3.) Press # Want to test your video connection? https://bluejeans.com/111 Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS Space Science & Engineering Center University of Wisconsin-Madison

1 0

Importance of bluefs fix in Octopus 15.2.10 ?
by Chris Palmer 19 Mar '21

19 Mar '21

When looking over the changelog for 15.2.10 I noticed some bluefs changes. One in particular caught my eye, and it was called out as a notable change: os/bluestore: fix huge reads/writes at BlueFS (pr#39701 <https://github.com/ceph/ceph/pull/39701>, Jianpeng Ma, Igor Fedotov) It wasn't obvious to me how important that one was for data integrity. We'll roll 15.2.10 onto our clusters shortly, but would appreciate an opinion on whether any delay in installing this would increase the risk of anything nasty (e.g. silent data corruption). If that was the case we will push the release through rather more urgently! Thanks, Chris

2 1

Re: v15.2.10 Octopus released
by David Galloway 18 Mar '21

18 Mar '21

Terribly sorry for the mistake. There was a bug in the script I use to sync packages to download.ceph.com that wasn't listing directories in the desired order. That meant the download.ceph.com/{rpm,deb}-octopus symlinks still pointed to 15.2.9. This is fixed. I'm re-running the container jobs to get those pushed too. On 3/18/21 10:45 AM, David Orman wrote: > Hi David, > > The "For Packages" link in your email/the blog posts do not appear to > work. Additionally, we browsed the repo, and it doesn't appear the > packages are uploaded, at least for debian-octopus: > http://download.ceph.com/debian-octopus/pool/main/c/ceph/. We only use > the release packages for cephadm bootstrapping, so it's not a > deal-breaker for us, just wanted to give you a head's up. > > Cheers, > David Orman > > On Thu, Mar 18, 2021 at 9:11 AM David Galloway <dgallowa(a)redhat.com> wrote: >> >> We're happy to announce the 10th backport release in the Octopus series. >> We recommend users to update to this release. For a detailed release >> notes with links & changelog please refer to the official blog entry at >> https://ceph.io/releases/v15-2-10-octopus-released >> >> Notable Changes >> --------------- >> >> * The containers include an updated tcmalloc that avoids crashes seen on >> 15.2.9. See `issue#49618 <https://tracker.ceph.com/issues/49618>`_ for >> details. >> >> * RADOS: BlueStore handling of huge(>4GB) writes from RocksDB to BlueFS >> has been fixed. >> >> * When upgrading from a previous cephadm release, systemctl may hang >> when trying to start or restart the monitoring containers. (This is >> caused by a change in the systemd unit to use `type=forking`.) After the >> upgrade, please run:: >> >> ceph orch redeploy nfs >> ceph orch redeploy iscsi >> ceph orch redeploy node-exporter >> ceph orch redeploy prometheus >> ceph orch redeploy grafana >> ceph orch redeploy alertmanager >> >> >> Getting Ceph >> ------------ >> * Git at git://github.com/ceph/ceph.git >> * Tarball at http://download.ceph.com/tarballs/ceph-15.2.10.tar.gz >> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ >> * Release git sha1: 27917a557cca91e4da407489bbaa64ad4352cc02 >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Email alerts from Ceph
by Andrew Walker-Brown 18 Mar '21

18 Mar '21

Hi all, How have folks implemented getting email or snmp alerts out of Ceph? Getting things like osd/pool nearly full or osd/daemon failures etc. Kind regards Andrew Sent from my iPhone

5 4

v15.2.10 Octopus released
by Dimitri Savineau 18 Mar '21

18 Mar '21

Hi, https://download.ceph.com/rpm-octopus/ symlink isn't updated to the new release [1] and the content is still 15.2.9 [2] As a consequence, the new Octopus container images 15.2.10 can't be built. [1] https://download.ceph.com/rpm-15.2.10/ [2] https://download.ceph.com/rpm-15.2.9/ Regards, Dimitri

1 0

v15.2.10 Octopus released
by David Galloway 18 Mar '21

18 Mar '21

We're happy to announce the 10th backport release in the Octopus series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-10-octopus-released Notable Changes --------------- * The containers include an updated tcmalloc that avoids crashes seen on 15.2.9. See `issue#49618 <https://tracker.ceph.com/issues/49618>`_ for details. * RADOS: BlueStore handling of huge(>4GB) writes from RocksDB to BlueFS has been fixed. * When upgrading from a previous cephadm release, systemctl may hang when trying to start or restart the monitoring containers. (This is caused by a change in the systemd unit to use `type=forking`.) After the upgrade, please run:: ceph orch redeploy nfs ceph orch redeploy iscsi ceph orch redeploy node-exporter ceph orch redeploy prometheus ceph orch redeploy grafana ceph orch redeploy alertmanager Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.10.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 27917a557cca91e4da407489bbaa64ad4352cc02

3 2

MON slow ops and growing MON store
by Janek Bevendorff 18 Mar '21

18 Mar '21

Hi, All of a sudden, we are experiencing very concerning MON behaviour. We have five MONs and all of them have thousands up to tens of thousands of slow ops, the oldest one blocking basically indefinitely (at least the timer keeps creeping up). Additionally, the MON stores keep inflating heavily. Under normal circumstances we have about 450-550MB there. Right now its 27GB and growing (rapidly). I tried restarting all MONs, I disabled auto-scaling (just in case) and checked the system load and hardware. I also restarted the MGR and MDS daemons, but to no avail. Is there any way I can debug this properly? I can’t seem to find how I can actually view what ops are causing this and what client (if any) may be responsible for it. Thanks Janek

4 13

PG export import
by Szabo, Istvan (Agoda) 18 Mar '21

18 Mar '21

Hi, I’ve tried to save some pg from a dead osd, I made this: Picked on the same server an osd which is not really used and stopped that osd and import the exported one from the dead one. root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-33 --no-mon-config --pgid 44.c0s0 --op export --file ./pg44c0s0 Exporting 44.c0s0 info 44.c0s0( empty local-lis/les=0/0 n=0 ec=192123/175799 lis/c=4865474/4851556 les/c/f=4865475/4851557/0 sis=4865493) Export successful root@server:~# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-34 --no-mon-config --op import --file ./pg44c0s0 get_pg_num_history pg_num_history pg_num_history(e5583546 pg_nums {20={173213=256},21={219434=64},22={220991=64},24={219240=32},25={1446965=128},42={175793=32},43={197388=64},44={192123=512}} deleted_pools ) Importing pgid 44.c0s0 write_pg epoch 4865498 info 44.c0s0( empty local-lis/les=0/0 n=0 ec=192123/175799 lis/c=4865474/4851556 les/c/f=4865475/4851557/0 sis=4865493) Import successful Started back 34 and it says the osd is running but in the cluster map it is down :/ root@server:~# systemctl status ceph-osd@34 -l ● ceph-osd(a)34.service<mailto:ceph-osd@34.service> - Ceph object storage daemon osd.34 Loaded: loaded (/lib/systemd/system/ceph-osd@.service<mailto:/lib/systemd/system/ceph-osd@.service>; enabled-runtime; vendor preset: enabled) Active: active (running) since Thu 2021-03-18 10:38:00 CET; 8min ago Process: 45388 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 34 (code=exited, sta> Main PID: 45392 (ceph-osd) Tasks: 60 Memory: 856.2M CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd(a)34.service └─45392 /usr/bin/ceph-osd -f --cluster ceph --id 34 --setuser ceph --setgroup ceph Mar 18 10:38:00 server systemd[1]: Starting Ceph object storage daemon osd.34... Mar 18 10:38:00 server systemd[1]: Started Ceph object storage daemon osd.34. Mar 18 10:38:21 server ceph-osd[45392]: 2021-03-18T10:38:21.817+0100 7f41738d5dc0 -1 osd.34 5583546 log_to_mon> Mar 18 10:38:21 server ceph-osd[45392]: 2021-03-18T10:38:21.825+0100 7f41738d5dc0 -1 osd.34 5583546 mon_cmd_ma> Any idea? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Same data for two buildings
by Denis Morejon Lopez 17 Mar '21

17 Mar '21

⁣I have a ceph cluster with 5 nodes. 3 in one building, and 2 in the other one. I put this information in the CRUSH. So that ceph were able to put one copy of objects in the nodes of one building and the other copy to the nodes of the other building. I mean, I setup replicas=2 in order to put the same information in both locations. But I know ceph cluster needs half +1 up nodes to keep one part of cluster working. I need at least a manual procedure to recover one of the two buildings if the other get down, or even if the link between them get down. I do not need 100% up, just something to block and unblock some nodes, and starts the two nodes if the building down were the one with 3 nodes.

2 1