- ceph-users - lists.ceph.io

Metadata pool space usage decreases
by Nathan MALO 02 Jun '23

02 Jun '23

Hello all, I have a weird behavior on my Cephfs. On May 29th I noticed a drop of 50Tb in my data pool. It has been followed by a decrease of space usage in the metadata pool since then. From May 29th, still happening as I write, the metadata pool has lost 1Tb over the initial 1.8Tb. Regarding the number of objects, it was 8.4 Million and is now 7.8 Million. I assume that my users have deleted a lot a of files on that day (our cephfs consists a very small files of about 4Mb). But such a huge decrease in metadata pool got me really concerned, it really seems huge. I thought that maybe it was the mds lazy deletion happening but I am not sure. Does anyone have any thought about this ? Do you know more about lazy deletion ? There is not much documentation about this online. Do you know any way, command, logfile to search in order to get the current lazy deletion operations ? I noticed that the read_ops, read_bytes, write_ops and write_bytes reported by the command rados df detail are negative for the metadata pool. My cluster is running Nautilus. Any help would be appreciated, Best regards, Nate

1 0

PGs incomplete - Data loss
by Benno Wulf 01 Jun '23

01 Jun '23

Hi guys, I'm awake since 36h and try to restore a broken ceph Pool (2 PGs incomplete) My vm are all broken. Some Boot, some Dont Boot... Also I have 5 removed disk with Data of that Pool "in my Hands" - Dont ask... So my question is it possible to restore Data of these other disks and "add" them thee others for healing? Best regards Ben

2 1

Cluster without messenger v1, new MON still binds to port 6789
by Robert Sander 01 Jun '23

01 Jun '23

Hi, a cluster has ms_bind_msgr1 set to false in the config database. Newly created MONs still listen on port 6789 and add themselves as providing messenger v1 into the monmap. How do I change that? Shouldn't the MONs use the config for ms_bind_msgr1? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

How to disable S3 ACL in radosgw
by Rasool Almasi 01 Jun '23

01 Jun '23

Hi, Is it possible to disable ACL in favor of bucket policy (on a bucket or globally)? The goal is to forbid users to use any bucket/object ACLs and only allow bucket policies. Seems there is no documentation in that regard which applies to Ceph RGW. Apology if I am sending this in the wrong mailing list. Regards, Rasool

1 0

cephadm does not honor container_image default value
by Daniel Krambrock 01 Jun '23

01 Jun '23

Hello. I think i found a bug in cephadm/ceph orch: Redeploying a container image (tested with alertmanager) after removing a custom `mgr/cephadm/container_image_alertmanager` value, deploys the previous container image and not the default container image. I'm running `cephadm` from ubuntu 22.04 pkg 17.2.5-0ubuntu0.22.04.3 and `ceph` version 17.2.6. Here is an example. Node clrz20-08 is the node altermanager is running on, clrz20-01 the node I'm controlling ceph from: * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` * Set alertmanager image ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` * Remove alertmanager image setting, revert to default: ``` root@clrz20-01:~# ceph config rm mgr mgr/cephadm/container_image_alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` -> `mgr/cephadm/container_image_alertmanager` is set to `quay.io/prometheus/alertmanager:v0.23.0`, but redeploy uses `quay.io/prometheus/alertmanager:latest`. This looks like a bug. * Set alertmanager image explicitly to the default value ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` -> Setting `mgr/cephadm/container_image_alertmanager` to the default setting fixes the issue. Bests, Daniel

3 3

How to specify to only build ceph-radosgw package from source?
by huy nguyen 31 May '23

31 May '23

Hi, I usually install the SRPM and then build from ceph.spec like this: rpmbuild -bb /root/rpmbuild/SPECS/ceph.spec --without ceph_test_package But it take a long time and contain many packages that I don't need. So is there a way to optimize this build process for only needed package, for example ceph-radosgw? Thanks.

1 0

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

BlueStore fragmentation woes
by Hector Martin 31 May '23

31 May '23

Hi, I've been seeing relatively large fragmentation numbers on all my OSDs: ceph daemon osd.13 bluestore allocator score block { "fragmentation_rating": 0.77251526920454427 } These aren't that old, as I recreated them all around July last year. They mostly hold CephFS data with erasure coding, with a mix of large and small files. The OSDs are at around 80%-85% utilization right now. Most of the data was written sequentially when the OSDs were created (I rsynced everything from a remote backup). Since then more data has been added, but not particularly quickly. At some point I noticed pathologically slow writes, and I couldn't figure out what was wrong. Eventually I did some block tracing and noticed the I/Os were very small, even though CephFS-side I was just writing one large file sequentially, and that's when I stumbled upon the free space fragmentation problem. Indeed, deleting some large files opened up some larger free extents and resolved the problem, but only until those get filled up and I'm back to fragmented tiny extents. So effectively I'm stuck at the current utilization, as trying to fill them up any more just slows down to an absolute crawl. I'm adding a few more OSDs and plan on doing the dance of removing one OSD at a time and replacing it with another one to hopefully improve the situation, but obviously this is going to take forever. Is there any plan for offering a defrag tool of some sort for bluestore? - Hector

7 27

[Pacific] Admin keys no longer works I get access denied URGENT!!!
by wodel youchi 31 May '23

31 May '23

Hi, After a wrong manipulation, the admin key no longer works, it seems it has been modified. My cluster is built using containers. When I execute ceph -s I get [root@controllera ceph]# ceph -s 2023-05-31T11:33:20.940+0100 7ff7b2d13700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2023-05-31T11:33:20.940+0100 7ff7b1d11700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] 2023-05-31T11:33:20.940+0100 7ff7b2512700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) From the log file I am getting : May 31 11:03:02 controllera docker[214909]: debug 2023-05-31T11:03:02.714+0100 7fcfc0c91700 0 cephx server client.admin: unexpected key: req.key=5fea877f2a68548b expected_key=8c2074e03ffa449a How can I recover the correct key? Regards.

2 1

all buckets mtime = "0.000000" after upgrade to 17.2.6
by alyarb＠gmail.com 31 May '23

31 May '23

I was running on 17.2.5 since October, and just upgraded to 17.2.6, and now the "mtime" property on all my buckets is 0.000000. On all previous versions going back to Nautilus this wasn't an issue, and we do like to have that value present. radosgw-admin has no quick way to get the last object in the bucket. Here's my tracker submission: https://tracker.ceph.com/issues/61264#change-239348

2 1