- ceph-users - lists.ceph.io

Re: cephadm: How to replace failed HDD where DB is on SSD
by Eugen Block 27 May '21

27 May '21

Could you share the output of lsblk -o name,rota,size,type from the affected osd node? My spec file is for a tiny lab cluster, in your case the db drive size should be something like '5T:6T' to specify a range. How large are the HDDs? Also maybe you should use the option 'filter_logic: AND', but I'm not sure if that's already the default, I remember that there were issues in Nautilus because the default was OR. I tried this just recently with a version similar to this, I believe it was 15.2.8 and it worked for me, but again, it's just a tiny virtual lab cluster. Zitat von Kai Stian Olstad <ceph+list(a)olstad.com>: > On 26.05.2021 11:16, Eugen Block wrote: >> Yes, the LVs are not removed automatically, you need to free up the >> VG, there are a couple of ways to do so, for example remotely: >> >> pacific1:~ # ceph orch device zap pacific4 /dev/vdb --force >> >> or directly on the host with: >> >> pacific1:~ # cephadm ceph-volume lvm zap --destroy /dev/<CEPH_VG>/<DB_LV> > > Thanks, > > I used the cephadm command and deleted the LV and the VG now has free space > > # vgs | egrep "VG|dbs" > VG #PV #LV #SN > Attr VSize VFree > ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 14 0 > wz--n- <5.24t 357.74g > > But it doesn't seams to be able to use it, because it can find anyting > > # ceph orch apply osd -i hdd.yml --dry-run > ################ > OSDSPEC PREVIEWS > ################ > +---------+------+-------------+----------+----+-----+ > |SERVICE |NAME |HOST |DATA |DB |WAL | > +---------+------+-------------+----------+----+-----+ > +---------+------+-------------+----------+----+-----+ > > I tried adding size as you have in your configuration > db_devices: > rotational: 0 > size: '30G:' > > Still it was unable to create the OSD. > > If I removed the : so it is 30GB exact size, it did find the disk, > but DB is not placed on a SSD since I do not have one with 30 GB > exact size > ################ > OSDSPEC PREVIEWS > ################ > +---------+------+-------------+----------+----+-----+ > |SERVICE |NAME |HOST |DATA |DB |WAL | > +---------+------+-------------+----------+----+-----+ > |osd |hdd |pech-hd-7 |/dev/sdt |- |- | > +---------+------+-------------+----------+----+-----+ > > > To me I looks like Cephadm can't use/find the free space on the VG > and use that as a new LV for the OSD. > > > -- > Kai Stian Olstad > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

best practice balance mode in HAproxy in front of RGW?
by Boris Behrens 27 May '21

27 May '21

Hello togehter, is there any best practive on the balance mode when I have a HAproxy in front of my rgw_frontend? currently we use "balance leastconn". Cheers Boris

2 4

Python lib usage access permissions
by Szabo, Istvan (Agoda) 27 May '21

27 May '21

Hi, Is there a way to be able to manage specific pools with the python lib without admin keyring? Not sure why it only works with admin keyring but not with the client keyring :/ Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

v15.2.13 Octopus released
by David Galloway 27 May '21

27 May '21

We're happy to announce the 13th backport release in the Octopus series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-13-octopus-released Notable Changes --------------- * RADOS: Ability to dynamically adjust trimming rate in the monitor and several other bug fixes. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-15.2.13.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: c44bc49e7a57a87d84dfff2a077a2058aa2172e2

1 0

cephfs vs rbd vs rgw
by Jorge Garcia 26 May '21

26 May '21

This may be too broad of a topic, or opening a can of worms, but we are running a CEPH environment and I was wondering if there's any guidance about this question: Given that some group would like to store 50-100 TBs of data on CEPH and use it from a linux environment, are there any advantages or disadvantages in terms of performance/ease of use/learning curve to using cephfs vs using a block device thru rbd vs using object storage thru rgw? Here are my general thoughts: cephfs - Until recently, you were not allowed to have multiple filesystems. Not sure about performance. rbd - Can only be mounted on one system at a time, but I guess that filesystem could then be served using NFS. rgw - A different usage model from regular linux file/directory structure. Are there advantages to forcing people to use this interface? I'm tempted to set up 3 separate areas and try them and compare the results, but I'm wondering if somebody has done some similar experiment in the past. Thanks for any help you can provide! Jorge

5 4

summarized radosgw size_kb_actual vs pool stored value doesn't add up
by Boris Behrens 26 May '21

26 May '21

Hi, I am still searching for a reason why these two values differ so much. I am currently deleting a giant amount of orphan objects (43mio, most of them under 64kb), but the difference get larger instead of smaller. This was the state two days ago: > > [root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc > 175977343264 > > [root@s3db1 ~]# rados df > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR > ... > eu-central-1.rgw.buckets.data 766 TiB 134632397 0 403897191 0 0 0 1076480853 45 TiB 532045864 551 TiB 0 B 0 B > ... > total_objects 135866676 > > [root@s3db1 ~]# ceph df... > eu-central-1.rgw.buckets.data 11 2048 253 TiB 134.63M 766 TiB 90.32 27 TiB And this is todays state: > > [root@s3db1 ~]# radosgw-admin bucket stats | grep '"size_kb_actual"' | awk '{ print $2 }' | tr -d , | paste -sd+ - | bc > 177144806812 > > [root@s3db1 ~]# rados df > ... > eu-central-1.rgw.buckets.data 786 TiB 120025590 0 360076770 > ... > total_objects 121261889 > > [root@s3db1 ~]# ceph df > ... > eu-central-1.rgw.buckets.data 11 2048 260 TiB 120.02M 786 TiB 92.59 21 TiB I would love to free up the missing 80TB :) Any suggestions? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 4

Pacific: _admin label does not distribute admin keyring
by Eugen Block 26 May '21

26 May '21

Hi, I'm currently exploring Pacific and the "_admin" label doesn't seem to work as expected. pacific1:~ # ceph -v ceph version 16.2.3-26-g422932e923 (422932e923eb429b9e16c352a663968f4b6f0a52) pacific (stable) According to the docs [1] the "_admin" label should instruct cephadm to distribute a ceph.conf and the admin keyring to the labeled host: > _admin: Distribute client.admin and ceph.conf to this host. > > By default, an _admin label is applied to the first host in the > cluster (where bootstrap was originally run), and the client.admin > key is set to be distributed to that host via the ceph orch > client-keyring ... function. Adding this label to additional hosts > will normally cause cephadm to deploy config and keyring files in > /etc/ceph. I can not confirm this, neither has the bootstrap node (pacific1) the "_admin" label (but he has the ceph.conf and admin keyring) nor does pacific2 have the admin keyring. Here's my host ls: pacific1:~ # ceph orch host ls HOST ADDR LABELS STATUS pacific1 192.168.124.183 mon pacific2 192.168.124.49 _admin mon pacific5 192.168.124.63 mon pacific1:~ # ssh pacific2 find /etc/ceph /etc/ceph /etc/ceph/rbdmap pacific1:~ # find /etc/ceph/ /etc/ceph/ /etc/ceph/ceph.pub /etc/ceph/ceph.conf /etc/ceph/rbdmap /etc/ceph/ceph.client.admin.keyring I searched a bit on tracker.ceph.com but couldn't find an existing issue. Is this worth a bug report? Regards, Eugen [1] https://docs.ceph.com/en/latest/cephadm/host-management/#special-host-labels

1 0

Ceph Pacific mon is not starting after host reboot
by Adrian Nicolae 25 May '21

25 May '21

Hi guys, I'm testing Ceph Pacific 16.2.4 in my lab before deciding if I will put it in production on a 1PB+ storage cluster with rgw-only access. I noticed a weird issue with my mons : - if I reboot a mon host, the ceph-mon container is not starting after reboot - I can see with 'ceph orch ps' the following output : mon.node01 node01 running (20h) 4m ago 20h 16.2.4 8d91d370c2b8 0a2e86af94b2 mon.node02 node02 running (115m) 12s ago 115m 16.2.4 8d91d370c2b8 51f4885a1b06 mon.node03 node03 stopped 4m ago 19h <unknown> <unknown> <unknown> (where node03 is the host which was rebooted). - I tried to start the mon container manually on node03 with '/bin/bash /var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run' and I've got the following output : debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 3314933069573799936, adjusting msgr requires debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112, adjusting msgr requires debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112, adjusting msgr requires debug 2021-05-23T08:24:25.192+0000 7f9a9e358700 0 mon.node03(a)-1(???).osd e408 crush map has features 432629308056666112, adjusting msgr requires cluster 2021-05-23T08:07:12.189243+0000 mgr.node01.ksitls (mgr.14164) 36380 : cluster [DBG] pgmap v36392: 417 pgs: 417 active+clean; 33 KiB data, 605 MiB used, 651 GiB / 652 GiB avail; 9.6 KiB/s rd, 0 B/s wr, 15 op/s debug 2021-05-23T08:24:25.196+0000 7f9a9e358700 1 mon.node03(a)-1(???).paxosservice(auth 1..51) refresh upgraded, format 0 -> 3 debug 2021-05-23T08:24:25.208+0000 7f9a88176700 1 heartbeat_map reset_timeout 'Monitor::cpu_tp thread 0x7f9a88176700' had timed out after 0.000000000s debug 2021-05-23T08:24:25.208+0000 7f9a9e358700 0 mon.node03@-1(probing) e5 my rank is now 1 (was -1) debug 2021-05-23T08:24:25.212+0000 7f9a87975700 0 mon.node03@1(probing) e6 removed from monmap, suicide. root@node03:/home/adrian# systemctl status ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service ● ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3(a)mon.node03.service - Ceph mon.node03 for c2d41ac4-baf5-11eb-865d-2dc838a337a3 Loaded: loaded (/etc/systemd/system/ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3@.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sun 2021-05-23 08:10:00 UTC; 16min ago Process: 1176 ExecStart=/bin/bash /var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.run (code=exited, status=0/SUCCESS) Process: 1855 ExecStop=/usr/bin/docker stop ceph-c2d41ac4-baf5-11eb-865d-2dc838a337a3-mon.node03 (code=exited, status=1/FAILURE) Process: 1861 ExecStopPost=/bin/bash /var/lib/ceph/c2d41ac4-baf5-11eb-865d-2dc838a337a3/mon.node03/unit.poststop (code=exited, status=0/SUCCESS) Main PID: 1176 (code=exited, status=0/SUCCESS) The only fix I could find was to redeploy the mon with : ceph orch daemon rm mon.node03 --force ceph orch daemon add mon node03 However, even if it's working after redeploy, it's not giving me a lot of trust to use it in a production environment having an issue like that. I could reproduce it with 2 different mons so it's not just an exception. My setup is based on Ubuntu 20.04 and docker instead of podman : root@node01:~# docker -v Docker version 20.10.6, build 370c289 Do you know a workaround for this issue or is this a known bug ? I noticed that there are some other complaints with the same behaviour in Octopus as well and the solution at that time was to delete the /var/lib/ceph/mon folder . Thanks.

5 10

Very uneven OSD utilization
by Sergei Genchev 25 May '21

25 May '21

Hello, I am running a nautilus cluster with 5 OSD nodes/90 disks that is exclusively used for S3. My disks are identical, but utilization ranges from 9% to 82%, and I am starting to get backfill_toofull errors even though I have only used 150TB out of 650TB of data. - Other than manually crush reweighting OSDs, is there any other option for me ? - what would cause this uneven distribution? Is there some documentation on how to track down what's going on? output of 'ceph osd df" is at https://pastebin.com/17HWFR12 Thank you!

2 2

rbd cp versus deep cp?
by Jan Kasprzak 25 May '21

25 May '21

Hello, Ceph users, what is the difference between "rbd cp" and "rbd deep cp"? What I need to do is to make a copy of the rbd volume one of our users inadveredly resized to a too big size, shrink the copied image to the expected size, verify that everything is OK, and then delete the original image. Would this work with rbd cp? Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2