- ceph-users - lists.ceph.io

High usage (DATA column) on dedicated for OMAP only OSDs
by Александр Руденко 24 Sep '24

24 Sep '24

Hi! We have S3/CephFS cluster with dedicated SSDs for bucket indexes, CephFS metadata and for a few small rgw metadata pools. We have special crush rules for these pools. All of our pools (pool naming is old because it's very old cluster) that are placed on SSDs: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 32 1.3 KiB 5 108 KiB 0 690 GiB .rgw.control 2 32 0 B 8 0 B 0 690 GiB .rgw 3 32 2.0 MiB 8.02k 328 MiB 0.02 690 GiB .rgw.gc 4 64 1.6 GiB 64 4.9 GiB 0.24 690 GiB .users.uid 5 32 1.8 MiB 2.63k 67 MiB 0 690 GiB .users 6 32 275 KiB 7.02k 379 MiB 0.02 690 GiB .usage 7 32 30 MiB 64 30 MiB 0 690 GiB .intent-log 8 64 0 B 0 0 B 0 690 GiB .log 9 64 6.6 GiB 34.29k 21 GiB 1.01 690 GiB .rgw.buckets.index 12 4096 1.9 TiB 248.96k 1.9 TiB 48.02 690 GiB .users.email 13 32 64 KiB 1.61k 72 MiB 0 690 GiB fs1_meta 14 64 467 MiB 35.39k 829 MiB 0.04 690 GiB But on all our SSDs we can see high DATA usage, for example: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 57 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 96 GiB 1.4 GiB 40 GiB 90.96 1.37 138 up osd.57 19 ssd 0.00005 1.00000 447 GiB 402 GiB 309 GiB 92 GiB 1.4 GiB 44 GiB 90.09 1.35 145 up osd.19 10 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 97 GiB 1.1 GiB 40 GiB 91.01 1.37 137 up osd.10 12 ssd 0.00005 1.00000 447 GiB 401 GiB 309 GiB 91 GiB 1.3 GiB 46 GiB 89.77 1.35 134 up osd.12 All SSDs are bluestore. Ceph 16.3.11 and many SSDs were deployed on 16.x. As I understand, the majority of these pools contain only "OMAP" which is stored in RocksDB and tracked in stats as *OMAP*. And I don't understand why we have such high *DATA* and we can see that DATA usage grows by 1-2 GB/day for the last 90 days (we don't have more monitoring data)! I have checked crush rules and all SSDs for PGs from other pools and can't see any other PGs from "pure" data pools. Rules are correct. How can I see what is stored in bluestore DATA on some OSD? I can export full RocksDB, but not DATA stored in bluestore.. What kind of DATA can be stored in these OSDs with these pools? We did a lot of offline compactions, 2-5 times for some SSDs in the last 90 days. But i'm not sure if it's related

4 14

Ceph Dashboard TLS
by matthew＠peregrineit.net 24 Sep '24

24 Sep '24

Hi All, I'm running an (experimental) 3-Node Ceph Reef (v18.2.4) Cluster. Each of the 3 nodes runs (amongst other services) the Ceph Dashboard - for fail-over purposes. I can connect to the Ceph Dashboard when not using TLS (ie ceph config set mgr mgr/dashboard/ssl false). I've got a private PKI (Step-CA) and I'd like to use its TLS Certificates for the Ceph Dashboard. The private PKI's CA Cert has been added to my browser, and internal websites, etc, using the internal PKI's Certificates work as expected. I'd like each of the three Ceph Nodes to use their own TLS Certificate. The 3 node's hostnames are ceph01.example.com, ceph02.example.com, and ceph03.example.com. The 3 node's mgr instances are (I think; & this could be where I'm going wrong) are mgr.ceph01, mgr.ceph02, and mgr.ceph03. I've created 3 TLS Certificates with names of cephdash-ceph01.example.com.crt, etc, CNs of cephdash-ceph01.example.com, etc, and SANs of ceph01.example.com, etc I've followed the documentation here: https://docs.ceph.com/en/reef/mgr/dashboard/#ssl-tls-support I have stopped and restarted the Ceph Dashboard service. Commands used (on ceph01.example.com) ceph dashboard set-ssl-certificate ceph01 -i cephdash-ceph01.example.com.crt ceph dashboard set-ssl-certificate-key ceph01 -i cephdash-ceph01.example.com.key ceph mgr module disable dashboard ceph mgr module enable dashboard It's not working (I get an Unable to connect page), and I can't work out why - but I suspect it'll be something bleedingly obvious. :-) My Qs: Can I use EC Certs or must I use RSA Certs? I've tryied both and neither work. Can anyone spot what I'm doing wrong? Thanks in advance Cheers Dulux-Oz

3 5

All new osds are made orphans
by Phil 24 Sep '24

24 Sep '24

Hi, on a healthy cluster, every osd creation made orphans osd. # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 26.38051 root default -7 3.63869 host A 1 hdd 3.63869 osd.1 up 1.00000 1.00000 -11 1.81940 host F 2 hdd 1.81940 osd.2 up 1.00000 1.00000 -5 7.27737 host J 3 hdd 3.63869 osd.3 up 1.00000 1.00000 5 hdd 3.63869 osd.5 up 1.00000 1.00000 -3 7.27737 host K 0 hdd 3.63869 osd.0 up 1.00000 1.00000 8 hdd 3.63869 osd.8 up 1.00000 1.00000 -9 6.36768 host S 6 hdd 3.63869 osd.6 up 1.00000 1.00000 7 hdd 2.72899 osd.7 up 1.00000 1.00000 4 0 osd.4 down 0 1.00000 9 0 osd.9 down 0 1.00000 neither 4 nor 9 osd appear in the crushmap (decompiled) Destroying and recreating osd recreate orphan osd. Any hint ?Thanks for all Best regardsPhilippe

2 3

CLT Meeting Notes 23 September 2024
by Casey Bodley 23 Sep '24

23 Sep '24

notes from today's meeting of the Ceph Leadership Team Squid 19.2.0 release ==================== Laura successfully built/signed prerelease packages LRC upgrade was the last step, but hit iscsi failures again * Ilya to create a tracker issue and tag Dan/Xiubo for details * Ilya to reschedule rbd/iscsi suite against 19.2.0 to determine whether this should block the release * release note needed * in infrastructure call, discuss future plan for LRC (probably nvmeof) Licensing discussion ==================== parent tracker https://tracker.ceph.com/issues/68082 since last week, trackers added for use of GPL in ppc https://tracker.ceph.com/issues/68086 and dashboard https://tracker.ceph.com/issues/68088 Ceph Executive Council ====================== regarding https://docs.ceph.com/en/latest/governance, the Ceph Steering Committee is overdue to elect a replacement member for the Ceph Executive Council we'll dedicate more time in next week's CLT to continue discussing the proposals and voting process Cephalocon ========== one more week until the deadline for early registration. update ceph.io with details about the Dev Summit

1 0

Overlapping Roots - How to Fix?
by Dave Hall 23 Sep '24

23 Sep '24

Helllo, I've reviewed some recent posts in this list and also searched Google for info about autoscale and overlapping roots. In what I have found I do not see anything that I can understand regarding how to fix the issue - probably because I don't deal with Crush on a regular basis. My particulars: I have an up-to-date Reef cluster in production with some EC pools and some replicated pools. My error messages look like: pool default.rgw.buckets.data won't scale due to overlapping roots: {-1, -2} From what I read and looking at 'ceph osd crush rule dump', it looks like the 8 replicated pools have "op": "take", "item": -1, "item_name": "default" whereas the 2 EC pools have "op": "take", "item": -2, "item_name": "default~hdd" To be sure, all of my OSDs are identical - HDD with SSD WAL/DB. Please advise on how to fix this. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

5 12

Radosgw bucket check fix doesn't do anything
by Reid Guyett 23 Sep '24

23 Sep '24

Hello, I recently moved a bucket from 1 cluster to another cluster using rclone. I noticed that the source bucket had around 35k objects and the destination bucket only had around 18k objects after the sync was completed. Source bucket stats showed: > radosgw-admin bucket stats --bucket mimir-prod | jq .usage > { > "rgw.main": { > "size": 4321515978174, > "size_actual": 4321552605184, > "size_utilized": 4321515978174, > "size_kb": 4220230448, > "size_kb_actual": 4220266216, > "size_kb_utilized": 4220230448, > "num_objects": 35470 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 66609, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 66, > "num_objects": 2467 > } > } Destination bucket stats showed: > radosgw-admin bucket stats --bucket mimir-prod | jq .usage > { > "rgw.main": { > "size": 4068176326491, > "size_actual": 4068212576256, > "size_utilized": 4068176326491, > "size_kb": 3972828444, > "size_kb_actual": 3972863844, > "size_kb_utilized": 3972828444, > "num_objects": 18525 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 108, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 1, > "num_objects": 4 > } > } > When I checked the source bucket using aws cli tool it showed around 18k objects. The bucket was actively being used so the 18k is slightly different. > aws --profile mimir-prod --endpoint-url https://my.objectstorage.domain > s3api list-objects --bucket mimir-prod > mimir_objs > cat mimir_objs | grep -c "Key" > 18090 I did a check on the source bucket and it showed a lot of invalid multipart objects. > radosgw-admin bucket check --bucket mimir-prod | head > { > "invalid_multipart_entries": [ > > "_multipart_network/01H9CFRA45MJWBHQRCHRR4JHV4/index.sJRTCoqiZvlge2cjz6gLU7DwuLI468zo.2", > > "_multipart_network/01HMCCRMTC5F4BFCZ56BKHTMWQ/index.6ypGbeMr6Jg3y7xAL8yrLL-v4sbFzjSA.3", > > "_multipart_network/01HMFKR56RRZNX9VT9B4F49MMD/chunks/000001.JIC7fFA_q96nal1yGXsVSPCY8EMe5AU8.2", > > "_multipart_network/01HMFKSND2E5BWF6QVTX8SDRRQ/index.57aSNeXn3j70H4EHfbNCD2RpoOp-P1Bv.2", > > "_multipart_network/01HMFKTDNA3FVSWW7N8KYY2C7N/chunks/000001.2~kRjRbLWWDf1e40P40LUzdU3f_x2P46Q.2", > > "_multipart_network/01HMFTMA8J1DEXYHKMVCXCC0GM/chunks/000001.GVajdCja0gHOLlgyFanF72A4B6ZqUpu5.2", > > "_multipart_network/01HMFTMA8J1DEXYHKMVCXCC0GM/chunks/000001.GYaouEePvEdbQosCb5jLFCAHrSm9VoDh.2", > > "_multipart_network/01HMFTMA8J1DEXYHKMVCXCC0GM/chunks/000001.r4HkP-JK-rBAWDoXBXKJJYEAjk39AswW.1", > ... > So I tried to run `radosgw-admin bucket check --check-objects --bucket mimir-prod --fix` and it showed that it was cleaning things with thousands of lines like > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.2 > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.3 > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.4 > 2024-09-17T12:19:42.212+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.5 > 2024-09-17T12:19:42.213+0000 7fea25b6f9c0 0 check_disk_state(): removing > manifest part from index: > mimir-prod:_multipart_tenant_prod/01J7Q778YXJXE23SRQZM9ZA4NH/chunks/000001.2~m6EI5fHFWxI-RmWB6TeFSupu7vVrCgh.6 but the end result shows nothing has changed. > "check_result": { > "existing_header": { > "usage": { > "rgw.main": { > "size": 4281119287051, > "size_actual": 4281159110656, > "size_utilized": 4281119287051, > "size_kb": 4180780554, > "size_kb_actual": 4180819444, > "size_kb_utilized": 4180780554, > "num_objects": 36429 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 66636, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 66, > "num_objects": 2468 > } > } > }, > "calculated_header": { > "usage": { > "rgw.main": { > "size": 4281119287051, > "size_actual": 4281159110656, > "size_utilized": 4281119287051, > "size_kb": 4180780554, > "size_kb_actual": 4180819444, > "size_kb_utilized": 4180780554, > "num_objects": 36429 > }, > "rgw.multimeta": { > "size": 0, > "size_actual": 0, > "size_utilized": 66636, > "size_kb": 0, > "size_kb_actual": 0, > "size_kb_utilized": 66, > "num_objects": 2468 > } > } > } > } > Does this command do anything? Is it the wrong command for this issue? How does one go about fixing buckets in this state? Thanks! Reid

2 8

Help with cephadm bootstrap and ssh private key location
by Kozakis, Anestis 23 Sep '24

23 Sep '24

Hi All, Very new to Ceph and hoping someone can help me out. We are implementing Ceph in our team's environment, and I have been able to manually set up a test cluster using cephadm bootstrap and answering all the prompts. What we want to do is to automate the setup and maintenance of the production Ceph cluster using SaltStack. This involves pre-configuring the hosts using Salt High States, including installation of packages, firewall configuration, etc etc, and the distribution of ssh keys (root user) and configuring openssh on each host. Whilst I can see the public ssh key in authotrized_keys on all the nodes, I cannot seem to find to find the private key on the initial admin host (the first node). It is not in /root/.ssh and the only file there is the authorized_keys file. Where does cephadm bootstrap store the private key? Also, if we already have ssh keys set up on all the hosts and we use the -skip-ssh option in the cephadm bootstrap command, will using a spec file with the --apply-spec option work to add additional hosts and all the services we need. Thanks in advance for any help/advice on this. Anestis Kozakis Systems Administrator - Multi-Level Security Solutions P: + 61 2 6122 0205 M: +61 4 88 376 339 anestis.kozakis(a)raytheon.com.au<mailto:anestis.kozakis@raytheon.com.au> Raytheon Australia Cybersecurity and Information Assurance 4 Brindabella Cct Brindabella Business Park Canberra Airport, ACT 2609 www.raytheonaustralia.com.au<http://www.raytheonaustralia.com.au/> LinkedIn<https://au.linkedin.com/company/raytheon-australia> | Twitter<https://twitter.com/RaytheonAU> | Facebook<https://www.facebook.com/RaytheonAustralia> | Instagram<https://www.instagram.com/raytheonaustralia/>

2 3

filter @ rados_object_list
by Daniel Biecker 23 Sep '24

23 Sep '24

Hi, I want to search for objects using the rados_object_list function. How do I set the filter? I always get error -22. What is the syntax for the filter value.

1 0

Directory missing in cephfs
by Pavel Kaygorodov 21 Sep '24

21 Sep '24

Hi, All! I'm running a small ceph cluster (16.2.3) with cephfs an number of replicas = 3. Recently, after one of the nodes failure, I have performed a number of copy/move (not a delete ones) operations, while on of the pgs was inactive. There was no hangs, no errors reported, but, some directories was disappeared in the directory, where it was moved. Now, all of pgs are active+clean (some of them are remapped), but missing directories are not returned. I have started journal recovery with the command: cephfs-journal-tool --rank files:all event recover_dentries summary it still working (more then 20 hours till now) without any visible results. I have made snapshots of data and metadata pools, before starting journal recovery, also made a backup of the journal. What is my next steps if the journal recovery will bring no results? Is there a way to manually find a specific directory record directly in the pool and restore it? I can write a program to fetch and parse cephs objects, but, may be, there is a specific tool for that? Thanks in advance, Pavel.

1 0

Multisite sync: is metadata transferred in plain text?
by Mary Zhang 21 Sep '24

21 Sep '24

Hi, We have a multisite Ceph configuration, with http (not https) sync endpoints. Are all sync traffic in plain text? We have concerns about metadata. For example, when syncing a newly created user and its access key and secret key from Master zone to a secondary zone, are the keys also in plain text? If so, what are options to encrypt it? Thank you, Mary

1 0