- ceph-users - lists.ceph.io

RFI: Prometheus, Etc, Services - Optimum Number To Run
by duluxoz 21 Jan '24

21 Jan '24

Hi All, In regards to the monitoring services on a Ceph Cluster (ie Prometheus, Grafana, Alertmanager, Loki, Node-Exported, Promtail, etc) how many instances should/can we run for fault tolerance purposes? I can't seem to recall that advice being in the doco anywhere (but of course, I probably missed it). I'm concerned about HA on those services - will they continue to run if the Ceph Node they're on fails? At the moment we're running only 1 instance of each in the cluster, but several Ceph Nodes are capable of running each - ie/eg 3 nodes configured but only count:1. This is on the latest version of Reef using cephadmin (if it makes a huge difference :-) ). So any advice, etc, would be greatly appreciated, including if we should be running any services not mentioned (not Mgr, Mon, OSD, or iSCSI, obviously :-) ) Cheers Dulux-Oz

2 1

rbd map snapshot, mount lv, node crash
by Marc 19 Jan '24

19 Jan '24

Am I doing something weird when I do on a ceph node (nautilus, el7): rbd snap ls vps-test -p rbd rbd map vps-test(a)vps-test.snap1 -p rbd mount -o ro /dev/mapper/VGnew-LVnew /mnt/disk <--- reset/reboot ceph node

2 1

How does mclock work?
by Frédéric Nass 19 Jan '24

19 Jan '24

Hello, Could someone please explain how mclock works regarding reads and writes? Does mclock intervene on both read and write iops? Or only on reads or only on writes? And what type of underlying hardware performance is calculated and considered by mclock? Seems to be only write performance. The mclock documentation shows HDDs and SSDs specific configuration options (capacity and sequential bandwidth) but nothing regarding hybrid setups and these configuration options do not distinguish reads and writes. But read and write performance are often not in par for a single drive and even less when using hybrid setups. With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads. With HDD only setups (RocksDB+WAL+Data on HDD), if mclock only considers write performance, the OSD may not take advantage of higher read performance. Can someone please shed some light on this? Best regards, Frédéric Nass Sous-direction Infrastructures et Services Direction du Numérique Université de Lorraine Tél : +33 3 72 74 11 35

11 14

Keyring location for ceph-crash?
by Jan Kasprzak 19 Jan '24

19 Jan '24

Hello, Ceph users, what is the correct location of keyring for ceph-crash? I tried to follow this document: https://docs.ceph.com/en/latest/mgr/crash/ # ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash' > /etc/ceph/ceph.client.crash.keyring and copy this file to all nodes. When I run ceph-crash.service on the node where client.admin.keyring is available, it works. But on the rest of nodes, it tries to access the admin keyring anyway. Journalctl -u ceph-crash says this: Jan 18 18:03:35 my.node.name systemd[1]: Started Ceph crash dump collector. Jan 18 18:03:35 my.node.name ceph-crash[2973164]: INFO:ceph-crash:pinging cluster to exercise our key Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.786+0100 7eff34016640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.786+0100 7eff34016640 -1 AuthRegistry(0x7eff2c063ce8) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.787+0100 7eff34016640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.787+0100 7eff34016640 -1 AuthRegistry(0x7eff2c067de0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.788+0100 7eff34016640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.788+0100 7eff34016640 -1 AuthRegistry(0x7eff340150c0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Jan 18 18:03:35 my.node.name ceph-crash[2973166]: [errno 2] RADOS object not found (error connecting to the cluster) Jan 18 18:03:35 my.node.name ceph-crash[2973164]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s Is the documentation outdated, or am I doing anything wrong? Thanks for any hint. This is non-containerized Ceph 18.2.1 on AlmaLinux 9. -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Indexless bucket constraints of ceph-rgw
by Jaemin Joo 19 Jan '24

19 Jan '24

I am considering Indexless bucket to increase the small file performance and maximum number of files using just HDDs. When I checked the constraints of indexless bucket in ceph-docs, it indicates the possible for bucket list problem, so I just know it makes the problem of versioning and sync. I don't understand what means of it. and I just guessed it's similar with db index feature. but it's not. Could you tell me the indexless bucket constraints?

1 0

Join us for the User + Dev Monthly Meetup - January 18th!
by Laura Flores 18 Jan '24

18 Jan '24

Hi Ceph users and developers, You are invited to join us at the User + Dev meeting this week Thursday, January 18th at 10:00 AM Eastern Time! See below for more meeting details. The focus topic, "Ceph Feature Request from the DKIST Data Center: Add a service backed by tape that is analogous to AWS Glacier", will be presented by Joel Davidow, a Ceph operator from the National Solar Observatory. In his talk, he will propose a feature request to add support for tape as a storage class with lifecycle management for object storage. Feel free to add questions or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time Conversions: UTC: Thursday, January 18, 15:00 UTC Mountain View, CA, US: Thursday, January 18, 7:00 PST Phoenix, AZ, US: Thursday, January 18, 8:00 MST Denver, CO, US: Thursday, January 18, 8:00 MST Huntsville, AL, US: Thursday, January 18, 9:00 CST Raleigh, NC, US: Thursday, January 18, 10:00 EST London, England: Thursday, January 18, 15:00 GMT Paris, France: Thursday, January 18, 16:00 CET Helsinki, Finland: Thursday, January 18, 17:00 EET Tel Aviv, Israel: Thursday, January 18, 17:00 IST Pune, India: Thursday, January 18, 20:30 IST Brisbane, Australia: Friday, January 19, 1:00 AEST Singapore, Asia: Thursday, January 18, 23:00 +08 Auckland, New Zealand: Friday, January 19, 4:00 NZDT -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 2

Wide EC pool causes very slow backfill?
by Torkil Svensgaard 18 Jan '24

18 Jan '24

Hi Our 17.2.7 cluster: " -33 886.00842 datacenter 714 -7 209.93135 host ceph-hdd1 -69 69.86389 host ceph-flash1 -6 188.09579 host ceph-hdd2 -3 233.57649 host ceph-hdd3 -12 184.54091 host ceph-hdd4 -34 824.47168 datacenter DCN -73 69.86389 host ceph-flash2 -5 252.27127 host ceph-hdd14 -2 201.78067 host ceph-hdd5 -81 288.26501 host ceph-hdd6 -31 264.56207 host ceph-hdd7 -36 1284.48621 datacenter TBA -77 69.86389 host ceph-flash3 -21 190.83224 host ceph-hdd8 -29 199.08838 host ceph-hdd9 -11 193.85382 host ceph-hdd10 -9 237.28154 host ceph-hdd11 -26 187.19536 host ceph-hdd12 -4 206.37102 host ceph-hdd13 " We recently created an EC 4+5 pool with failure domain datacenter. The DCN datacenter only had 2 hdd hosts so we added one more to make it possible at all, since each DC needs 3 shards, as I understand it. Backfill was really slow though, so we just added another host to the DCN datacenter. Backfill looks like this: " data: volumes: 1/1 healthy pools: 13 pools, 11153 pgs objects: 311.53M objects, 1000 TiB usage: 1.6 PiB used, 1.6 PiB / 3.2 PiB avail pgs: 60/1669775060 objects degraded (0.000%) 373356926/1669775060 objects misplaced (22.360%) 5944 active+clean 5177 active+remapped+backfill_wait 22 active+remapped+backfilling 4 active+recovery_wait+degraded+remapped 3 active+recovery_wait+remapped 2 active+recovery_wait+degraded 1 active+recovering+degraded+remapped io: client: 73 MiB/s rd, 339 MiB/s wr, 1.06k op/s rd, 561 op/s wr recovery: 1.2 GiB/s, 313 objects/s " Given that the first host added had 19 OSDs, with none of them anywhere near the target capacity, and the one we just added has 22 empty OSDs, having just 22 PGs backfilling and 1 recovering seems somewhat underwhelming. Is this to be expected with such a pool? Mclock profile is high_recovery_ops. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 6

Debian 12 (bookworm) / Reef 18.2.1 problems
by Chris Palmer 18 Jan '24

18 Jan '24

I was delighted to see the native Debian 12 (bookworm) packages turn up in Reef 18.2.1. We currently run a number of ceph clusters on Debian11 (bullseye) / Quincy 17.2.7. These are not cephadm-managed. I have attempted to upgrade a test cluster, and it is not going well. Quincy only supports bullseye, and Reef only supports bookworm, we are reinstalling from bare metal. However I don't think either of these two problems are related to that. Problem 1 -------------- A simple "apt install ceph" goes most of the way, then errors with Setting up cephadm (18.2.1-1~bpo12+1) ... usermod: unlocking the user's password would result in a passwordless account. You should set a password with usermod -p to unlock this user's password. mkdir: cannot create directory ‘/home/cephadm/.ssh’: No such file or directory dpkg: error processing package cephadm (--configure): installed cephadm package post-installation script subprocess returned error exit status 1 dpkg: dependency problems prevent configuration of ceph-mgr-cephadm: ceph-mgr-cephadm depends on cephadm; however: Package cephadm is not configured yet. dpkg: error processing package ceph-mgr-cephadm (--configure): dependency problems - leaving unconfigured The two cephadm-related packages are then left in an error state, which apt tries to continue each time it is run. The cephadm user has a login directory of /nonexistent, however the cephadm --configure script is trying to use /home/cephadm (as it was on Quincy/bullseye). So, we aren't using cephadm, and decided to keep going as the other packages were actually installed, and deal with the package state later. Problem 2 --------------- I upgraded 2/3 monitor nodes without any other problems, and (for the moment) removed the other Quincy monitor prior to rebuild. I then shutdown the remaining Quincy manager, and attempted to start the Reef manager. Although the manager is running, "ceph mgr services" shows it is only providing the restful and not the dashboard service. The log file has lots of the following error: ImportError: PyO3 modules may only be initialized once per interpreter process and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process Questions --------------- 1. Have the Reef/bookworm packages ever been tested in a non-cephadm environment? 2. I want to revert this cluster back to a fully functional state. I cannot bring back up the remaining Quincy monitor though ("require release 18 > 17"). Would I have to go through the procedure of starting over, and trying to rescue the monmap from the OSDs? (OSDs and an active MDS are still up and running Quincy). I'm aware that process exists but have never had to delve into it. Thanks, Chris

3 5

Adding OSD's results in slow ops, inactive PG's
by Ruben Vestergaard 18 Jan '24

18 Jan '24

Hi We have a cluster with which currently looks like so: services: mon: 5 daemons, quorum lazy,jolly,happy,dopey,sleepy (age 13d) mgr: jolly.tpgixt(active, since 25h), standbys: dopey.lxajvk, lazy.xuhetq mds: 1/1 daemons up, 2 standby osd: 449 osds: 425 up (since 15m), 425 in (since 5m); 5104 remapped pgs data: volumes: 1/1 healthy pools: 13 pools, 11153 pgs objects: 304.11M objects, 988 TiB usage: 1.6 PiB used, 1.4 PiB / 2.9 PiB avail pgs: 6/1617270006 objects degraded (0.000%) 366696947/1617270006 objects misplaced (22.674%) 6043 active+clean 5041 active+remapped+backfill_wait 66 active+remapped+backfilling 2 active+recovery_wait+degraded+remapped 1 active+recovering+degraded It's currently rebalancing after adding a node, but this rebalance has been rather slow -- right now it's running 66 backfills, but it seems to stabilize around 8 backfills eventually. We figured that perhaps adding another node might speed things up. Immediately upon adding the node, we get slow ops and inactive PG's. Removing the new node gets us back in working order. It turns out that even adding 1 OSD breaks the cluster, and immediately sends it here: [WRN] PG_DEGRADED: Degraded data redundancy: 6/1617265712 objects degraded (0.000%), 3 pgs degraded pg 37.c8 is active+recovery_wait+degraded+remapped, acting [410,163,236,209,7,283,155,143,78] pg 37.1a1 is active+recovering+degraded, acting [234,424,163,74,22,128,177,153,181] pg 37.1da is active+recovery_wait+degraded+remapped, acting [163,408,230,190,93,284,50,78,44] [WRN] SLOW_OPS: 22 slow ops, oldest one blocked for 54 sec, daemons [osd.11,osd.110,osd.112,osd.117,osd.120,osd.123,osd.13,osd.136,osd.144,osd.157]... have slow ops. The OSD added had number 431, so it does not appear to be the immediate cause of the slow ops, however, removing 431 immediately clears the problem. We thought we might be experiencing 'Crush giving up too soon' symptoms [1], as we have seen similar behaviour on another pool, but it does not appear to be the case here. We went through the motions described on the page and everything looked OK. At least one pool which stops working is a 4+2 EC pool, placed on spinning rust, some 200-ish disks distributed across 13 nodes. I'm not sure if other pools break, but that particular 4+2 EC pool is rather important so I'm a little wary of experimenting blindly. Any thoughts on where to look next? Thanks, Ruben Vestergaard [1] https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#cru…

4 9

minimal permission set for an rbd client
by cek+ceph＠deepunix.net 18 Jan '24

18 Jan '24

I'm following the guide @ https://docs.ceph.com/en/latest/rbd/rados-rbd-cmds/ but I'm not following why would an `mgr` permission be required to have a functioning RBD client? Thanks.

2 1