June 2023 - ceph-users - lists.ceph.io

Tuning RGW rgw_object_stripe_size and rgw_max_chunk_size
by David Oganezov 14 Jun '23

14 Jun '23

Hello! In my company we are using an EC 8+4 pool backed by HDDs. Since the default rgw_max_chunk_size and rgw_object_stripe_size parameters are set to 4MiB, each HDD's chunk size is 512KiB, which is not ideal. I was wondering if anyone tried playing with this parameters, as increasing them to 32MiB for example, can theoretically double the throughput of the cluster, as each HDD's chunk will be at 4MiB, which is more suited to HDDs. Thanks!

1 0

Upgrading standard Debian packages
by Ben Thompson 14 Jun '23

14 Jun '23

Hi Everyone, I am new to ceph and looking at how to upgrade a Debian 10 cluster to Debian 11. Each node is running the stock Debian (Luminous) packages and the upgrade to Bullseye (Debian 11) will pull in Nautilus. According to the documentation here https://docs.ceph.com/en/latest/releases/nautilus/#upgrading-from-mimic-or-… the procedure is to upgrade each daemon one by one (MON -> MGR -> OSD). However, it seems that the Debian packages are all interdependent on ceph-base which means that upgrading one at a time is not possible. I would be grateful for any advice. -- Ben Thompson

1 0

Updating the Grafana SSL certificate in Quincy
by Thorne Lawler 14 Jun '23

14 Jun '23

Hi everyone! I have a containerised (cephadm built) 17.2.6 cluster where I have installed a custom commercial SSL certificate under dashboard. Before I upgraded from 17.2 to 17.2.6, I successfully installed the custom SSL cert everywhere, including grafana, but since the upgrade I am finding that I can't update the certificate for grafana. Have tried many many commands like the following: ceph config-key set mgr/cephadm/grafana_crt -i /etc/pki/tls/certs/_.quick.net.au_2024.pem ceph orch reconfig grafana ceph dashboard set-grafana-frontend-api-url https://san.quick.net.au:3000 restorecon /etc/pki/tls/certs/_.quick.net.au_2024.pem ceph orch reconfig grafana ceph dashboard set-grafana-frontend-api-url https://san.quick.net.au:3000 ceph dashboard set-grafana-frontend-url https://san.quick.net.au:3000 ceph dashboard grafana ceph dashboard grafana dashboards update ceph orch reconfig grafana ceph config-key set mgr/cephadm/grafana_crt -i /etc/pki/tls/certs/_.quick.net.au_2024.pem ceph orch redeploy grafana ceph config set mgr mgr/dashboard/GRAFANA_API_URL https://san.quick.net.au:3000 ...but to no avail. The grafana frames within dashboard continue to use the self-signed key. Have the commands for updating this changed between 17.2.0 and 17.2.6? Thank you. -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

2 2

How to release the invalid tcp connection under radosgw?
by Louis Koo 14 Jun '23

14 Jun '23

connections: [root@et-uos-warm02 deeproute]# netstat -anltp | grep rados | grep 10.x.x.x:7480 | grep ESTAB | grep 10.12 | wc -l 6650 The prints: tcp 0 0 10.x.x.x:7480 10.x.x.12:40210 ESTABLISHED 76749/radosgw tcp 0 0 10.x.x.x:7480 10.x.x.12:33218 ESTABLISHED 76749/radosgw tcp 0 0 10.x.x.x:7480 10.x.x.12:33546 ESTABLISHED 76749/radosgw tcp 0 0 10.x.x.x:7480 10.x.x.12:50024 ESTABLISHED 76749/radosgw .... but client ip 10.x.x.12 is unreachable(because the node was shutdown), the status of the tcp connections is always "ESTABLISHED", how to fix it?

2 3

log file timestamp precision
by alienfirstzen＠yahoo.com.tw 14 Jun '23

14 Jun '23

Hi,In my ceph log file, the timestamp shows only to millisecond, like below. How to make it show microsecond and nanosecond? By config file, or need to modify source code and where? 2023-06-10T09:38:50.549+0800 7f639d009040 15 client.1500594.objecter _session_op_assign 8 1... My ceph version is ceph:v16 docker image Many thanks!

1 0

Ceph User + Dev Monthly June Meetup
by Neha Ojha 13 Jun '23

13 Jun '23

Hi everyone, This month's Ceph User + Dev Monthly meetup is on June 15, 14:00-15:00 UTC. We'd love to share details about the first Reef release candidate. Please feel free to add more topics to https://pad.ceph.com/p/ceph-user-dev-monthly-minutes. Hope to see you there! Thanks, Neha

1 0

First Reef release candidate - v18.1.0
by Neha Ojha 13 Jun '23

13 Jun '23

Hi everyone, This is the first release candidate for Reef. The Reef release comes with a new RockDB version (7.9.2) [0], which incorporates several performance improvements and features. Our internal testing doesn't show any side effects from the new version, but we are very eager to hear community feedback on it. This is the first release to have the ability to tune RockDB settings per column family [1], which allows for more granular tunings to be applied to different kinds of data stored in RocksDB. A new set of settings has been used in Reef to optimize performance for most kinds of workloads with a slight penalty in some cases, outweighed by large improvements in use cases such as RGW, in terms of compactions and write amplification. We would highly encourage community members to give these a try against their performance benchmarks and use cases. The detailed list of changes in terms of RockDB and BlueStore can be found in https://pad.ceph.com/p/reef-rc-relnotes. If any of our community members would like to help us with performance investigations or regression testing of the Reef release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/reef_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. Overall things are looking pretty good based on our testing. Please try it out and report any issues you encounter. Happy testing! Thanks, Neha Get the release from * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: c2214eb5df9fa034cc571d81a32a5414d60f0405 [0] https://github.com/ceph/ceph/pull/49006 [1] https://github.com/ceph/ceph/pull/51821

1 0

RGW striping configuration.
by Teja A 13 Jun '23

13 Jun '23

Hello I am working on an application that uses rados gateway to store objects onto a ceph cluster. I am currently working on optimizing the latency for storing/retrieving objects from the cluster. My goal to improve read/write latencies is to have RGW write/read multiple rados objects in parallel as described here <https://docs.ceph.com/en/latest/architecture/#data-striping> - "Significant write performance occurs when the client writes the stripe units to their corresponding objects in parallel.". Just like with RAID0, by having a large number of rados objects that each radosgw object gets mapped to, we can achieve lower latencies as we are not bound by the throughput of a single disk. That documentation suggests that we can configure the stripe count as well as stripe width which would let us indirectly control how many rados objects each radosgw object gets mapped to. I want to be able to change these parameters and run benchmarks against my pools. The key parameter I am therefore interested in controlling is the stripe count (i.e. the number of distinct objects each radosgw object is mapped to). More specifically, in the diagram <https://docs.ceph.com/en/latest/_images/ditaa-96a6fc80dad17fb53f161987ed64f…> attached to those docs, I see that the stripe_count is 4 (4 rados objects being written to for a single RGW object). I want to be able to experiment with varying numbers for that stripe_count. I am having trouble figuring out what configuration parameters exist in radosgw that lets me control this. I see that there is a stripe_width <https://github.com/ceph/ceph/blob/714cdc4e8767a153f825e857efdc28bb481528a1/…> and a rgw_max_chunk_size <https://docs.ceph.com/en/latest/radosgw/config-ref/#confval-rgw_max_chunk_s…> but did not find anything for stripe_count. Configuring the stripe_width alone is not sufficient as I would need to set the stripe_unit size as well to get the desired number for stripe_count, but I did not find either. Am I understanding that correctly? If so, can someone please point me to where and how this configuration should be set? I appreciate your help!

2 2

RGW: exposing multi-tenant
by Rok Jaklič 13 Jun '23

13 Jun '23

Hi, are there any drawbacks of exposing multi-tenant deployment of RGWs directly to users so they can use any S3 client to connect to service or should we put something in front of RGWs? How many users in multi-tenant deployment can CEPH handle? Kind regards, Rok

1 0

Announcing go-ceph v0.22.0
by Anoop C S 13 Jun '23

13 Jun '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.22.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S

1 0

ceph fs perf stats output is empty
by Denis Polom 13 Jun '23

13 Jun '23

Hi I'm running latest Ceph Pacific 16.2.13 with Cephfs. I need to collect performance stats per client, but getting empty list without any numbers I even run dd on client against mounted ceph fs, but output is only like this: #> ceph fs perf stats 0 4638 192.168.121.1 {"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", " opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metada ta_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {}, "global_metrics": {}, "metrics": {"delayed_ranks": []}} Do I need to set some extra options? Does it work for some of you guys? Thank you dp

3 4

Monitor mkfs with keyring failed
by Michel Niyoyita 12 Jun '23

12 Jun '23

Hello Team , I am trying to make a ceph cluster with 3 nodes running ubuntu 20.04 and is configured using ceph-ansible , because it is a testing cluster OSD servers are same which will run as monitors . while installation I am facing following error : TASK [ceph-mon : ceph monitor mkfs with keyring] ************************************************************************************************* Monday 12 June 2023 15:48:18 +0000 (0:00:00.089) 0:01:20.254 *********** fatal: [ceph-osd1]: FAILED! => changed=true cmd: - ceph-mon - --cluster - ceph - --setuser - ceph - --setgroup - ceph - --mkfs - -i - ceph-osd1 - --fsid - d80c5efc-1d3e-4054-a6a4-3855dddd3543 - --keyring - /var/lib/ceph/tmp/ceph.mon..keyring delta: '0:00:00.282151' end: '2023-06-12 15:48:19.062350' msg: non-zero return code rc: -6 start: '2023-06-12 15:48:18.780199' stderr: |- /build/ceph-16.2.13/src/mon/MonMap.h: In function 'void MonMap::add(const mon_info_t&)' thread 7ffae5cb8700 time 2023-06-12T15:48:18.810345+0000 if you face this before Kindly help to solve . Regards Michel

1 0

bucket notification retries
by Yuval Lifshitz 12 Jun '23

12 Jun '23

Dear Community, I would like to collect your feedback on this issue. This is a followup from a discussion that started in the RGW refactoring meeting on 31-May-23 (thanks @Krunal Chheda <kchheda3(a)bloomberg.net> for bringing up this topic!). Currently persistent notifications are retried indefinitely. The only limiting mechanism that exists is that all notifications to a specific topic are stored in one RADOS object (of size 128MB). Assuming notifications are ~1KB at most, this would give us at least 128K notifications that can wait in the queue. When the queue fills up (e.g. kafka broker is down for 20 minutes, we are sending ~100 notifications per second) we start sending "slow down" replies to the client, and in this case the S3 operation will not be performed. This means that, for example, an outage of the kafka system would eventually cause an outage of our service. Note that this may also be a result of a misconfiguration of the kafka broker, or decommissioning of a broker. To avoid that, we propose several options: * use a fifo instead of a queue. This would allow us to hold more than 128K messages - survive longer broker outages, and at a higher message rate. there should still probably be a limit set on the size of the fifo * define maximum number of retries allowed for a notification * define maximum time the notification may stay in the queue before it is removed We should probably start with these definitions done as topic attributes, reflecting our delivery guarantees for this specific destination. Will try to capture the results of the discussion in this tracker: https://tracker.ceph.com/issues/61532 Thanks, Yuval

2 3

stray daemons not managed by cephadm
by farhad kh 12 Jun '23

12 Jun '23

i deployed the ceph cluster with 8 node (v17.2.6) and after add all of hosts, ceph create 5 mon daemon instances i try decrease that to 3 instance with ` ceph orch apply mon --placement=label:mon,count:3 it worked, but after that i get error "2 stray daemons not managed by cephadm" . But every time I tried to deploy and delete other instances, this number increased Now I have 7 daemon that are not managed by cephadm How to deal with this issue? ------------------------ [root@opcsdfpsbpp0201 ~]# ceph -s cluster: id: 79a2627c-0821-11ee-a494-00505695c58c health: HEALTH_WARN 16 stray daemon(s) not managed by cephadm services: mon: 3 daemons, quorum opcsdfpsbpp0201,opcsdfpsbpp0205,opcsdfpsbpp0203 (age 2m) mgr: opcsdfpsbpp0201.vttwxa(active, since 27h), standbys: opcsdfpsbpp0207.kzxepm mds: 1/1 daemons up, 2 standby osd: 74 osds: 74 up (since 26h), 74 in (since 26h) data: volumes: 1/1 healthy pools: 6 pools, 6 pgs objects: 2.10k objects, 8.1 GiB usage: 28 GiB used, 148 TiB / 148 TiB avail pgs: 6 active+clean io: client: 426 B/s rd, 0 op/s rd, 0 op/s wr [root@opcsdfpsbpp0201 ~]# ceph health detail HEALTH_WARN 16 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 16 stray daemon(s) not managed by cephadm stray daemon mon.opcsdfpsbpp0207 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0211 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0203 not managed by cephadm stray daemon mon.opcsdfpsbpp0207 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0211 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0205 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0207 not managed by cephadm stray daemon mon.opcsdfpsbpp0207 on host opcsdfpsbpp0209 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0209 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0211 not managed by cephadm stray daemon mon.opcsdfpsbpp0215 on host opcsdfpsbpp0211 not managed by cephadm stray daemon mon.opcsdfpsbpp0211 on host opcsdfpsbpp0213 not managed by cephadm stray daemon mon.opcsdfpsbpp0209 on host opcsdfpsbpp0215 not managed by cephadm stray daemon mon.opcsdfpsbpp0213 on host opcsdfpsbpp0215 not managed by cephadm

3 2

Keepalived configuration with cephadm
by Luis Domingues 12 Jun '23

12 Jun '23

Hi all, We are running 1 test cluster ceph with cephadm. Currently last pacific (16.2.13). We use cephadm to deploy keepalived:2.1.5 and HAProxy:2.3. We have 3 VIPs, 1 for each instance of HAProxy. But, we do not use the same network for managing the cluster and for the public traffic. We have a management network to connect to the machines, and for cephadm to do the deployments, and a prod network where the connections to HAproxy will be done. Our spec file looks like: --- service_type: ingress service_id: rgw.rgw placement: label: rgws spec: backend_service: rgw.rgw virtual_ips_list: - 10.X.X.10/24 - 10.X.X.2/24 - 10.X.X.3/24 frontend_port: 443 monitor_port: 1967 Our issue is that cephadm will populate `unicast_src_ip` and `unicast_peer` using the IPs from mgmt network and not the ones from prod network. A quick look into the code and it seems to be design that way. Our issue is that doing so, Keepalived instances will not talk to each other because the VRRP traffic is only allowed on our prod network. I quicky tested removing `unicast_src_ip` and `unicast_peer` and keepalived instances where able to talk to each other. My question, did I missed something on the configuration? Or should we add some kind of option to generate keepalived's config without `unicast_src_ip` and `unicast_peer`? Thanks, Luis Domingues Proton AG

1 0

The num of objects with cmd "rados -p xxx" not equal with s3 api?
by Louis Koo 12 Jun '23

12 Jun '23

List objects with rados cmd: "rados -p oss.rgw.buckets.index ls | grep "c2af65dc-b456-4f5a-be6a-2a142adeea75.335721.1" | awk '{print "rados listomapkeys -p oss.rgw.buckets.index "$1 }'|sh -x" some objects like this, I don't know what's it "?1000_DR-MON-1_20220307_224723/configs/lidars.cfgiUFWot6m-OJzb4cl6fC4czRzv9tGJMLw ?1000_DR-MON-1_20220307_224723/logs/sentry/DR_MON_1_20220307_224724_carstatus.jsoniYf0RYRvfYXWwwr6U8JDQ59ZSN2SLXg5 ?1000_MONDEO_A12/DR-MKZ-1_20220114_191640/coredumps/i.b2Je.JKFrSJAmlga7IIXLyKFSmXgAo ?1000_MONDEO_A12/DR-MKZ-1_20220114_191640/coredumps/iCuATmyPL9vXXfsSHENNkIpAqS.EZKFj ?1000_MONDEO_A12/DR-MKZ-1_20220211_190901/logs/gnss_adapter/i7g.vodrYiNsy4ljwy6UsQNptRHoY4Uz ?1000_MONDEO_A12/DR-MKZ-1_20220211_190901/logs/gnss_adapter/izHgzCRwJY2eGqV0ifuz9DaS50p9zmjA ?1000_MONDEO_A12/DR-MON-1_20211002_012629/config/iEYp.8-exWsChWoEGKTPBRzfFcm8EmK4 ?1000_MONDEO_A12/DR-MON-1_20211005_011823/log_other/iikTrMOJno2pKZFzxhBlJ04TqTrv37Be ?1000_MONDEO_A12/DR-MON-1_20211019_181039/task_information/iiyyJUW86YITRf023drwHV5TBNlxCIwJ ?1000_MONDEO_A12/DR-MON-1_20211019_181039/task_information/int4GOdWMZ7zXddTVtChac1TlbgwgJpJ ?1000_MONDEO_A12/DR-MON-1_20211110_235107_OT1495/config/irVFxhMgTrTI.cRhUPdt3YxMg-pX.GVy ?1000_MONDEO_A12/DR-MON-1_20211110_235107_OT1495/config/ixGn1hqiDGPR1IOmhW6aCGLJPELJAHNW ?1000_MONDEO_A12/DR-MON-1_20211130_230853/bag/important/ia7X4HvOe4VTk6i6jrgOLpAGzvUDlCpz ?1000_MONDEO_A12/DR-MON-1_20211130_230853/bag/important/ilyff1HsUSYEuWNr4smOpfS2lDyY.CoZ ?1000_MONDEO_A12/DR-MON-1_20211130_230853/task_information/task_description.jsoniDMuGFwTNPY2fTKuDIqSlVfy5AY6AydN ?1000_MONDEO_A12/DR-MON-1_20211203_203345/bag/bags_info.jsoniBEU6igyLo59VwAijFYQVl.iJMUP5s6F ?1000_MONDEO_A12/DR-MON-1_20211213_233958/config/lidars.cfgi.vISYc4d7gOj-qv2grSBhSZJ4LAPIti ?1000_MONDEO_A12/DR-MON-1_20211214_001357/task_information/sentry_config.jsonio8sOurJMZZReHUMu4jx116dz2buWwS0 ?1000_MONDEO_A12/DR-MON-1_20211215_195308/bag/important/iCOoK1O1ydDYHgDQsXAfiCdKc84gglEb ?1000_MONDEO_A12/DR-MON-1_20211215_195308/bag/important/iTQxsDHPef1UGyMmgQL9RNQBlxLHEHyT ?1000_MONDEO_A12/DR-MON-1_20211220_223504/log/DR_MON_1_20211220_223504_syslog.jsonig-E8J2jUe2IekrF8EIOebxaC3HZHWeF ?1000_MONDEO_A12/DR-MON-1_20211220_223504/log/DR_MON_1_20211220_223504_syslog.jsoniw2cN-1wvaz35.fkanuE.zEke95.UZ8V ?1000_MONDEO_A12/DR-MON-1_20211228_200413/log_other/gnss_imu_time.logiCL3l2NTALD2Kyi-KbxYAP4oQg.m6qjN ?1000_MONDEO_A12/DR-MON-1_20211228_200413/log_other/gnss_imu_time.logitAPux7n2MeOIGLejTDnCyirEpaQoQuQ ?1000_MONDEO_A12/DR-MON-1_20220104_195217/config/versioni0QqTxQh7EdOCgTIU3KHgaZfMXiAAQXF ?1000_MONDEO_A12/DR-MON-1_20220104_195217/config/versionida0hUxK8TREImCjsG" total objects: 395797 List obejects with s3 api: total objects: 67498

1 0

what are the options for config a CephFS client session
by Denis Polom 12 Jun '23

12 Jun '23

Hi, I didn't find any doc and any way how to get to know valid options to configure client session over mds socket: #> ceph tell mds.mds1 session config session config <client_id:int> <option> [<value>] : Config a CephFS client session Any hint on this? Thank you

2 1

Container image of Pacific latest version
by mahnoosh shahidi 12 Jun '23

12 Jun '23

Hi all, It seems the latest Pacific image in the registry is 16.2.11. Is there any plan to push the latest version of Pacific (16.2.13) in near future? Best Regards, Mahnoosh

3 4

Operations: cannot update immutable features
by Adam Boyhan 12 Jun '23

12 Jun '23

I have a small cluster on Pacific with roughly 600 RBD images. Out of those 600 images I have 2 which are in a somewhat odd state. root@cephmon:~# rbd info Cloud-Ceph1/vm-134-disk-0 rbd image 'vm-134-disk-0': size 1000 GiB in 256000 objects order 22 (4 MiB objects) snapshot_count: 11 id: 7c326b8b4567 block_name_prefix: rbd_data.7c326b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations op_features: snap-trash flags: create_timestamp: Fri Aug 14 07:11:44 2020 access_timestamp: Thu Jun 8 06:31:06 2023 modify_timestamp: Thu Jun 8 06:31:11 2023 Specifically, the feature "operations". I never set this feature; and I don't see it listed in documentation. This feature is causing my backup software to not be able to backup the rdb. Otherwise, it's working aok. I did attempt to remove the feature. root@cephmon:~# rbd feature disable Cloud-Ceph1/vm-464-disk-0 operations rbd: failed to update image features: 2023-06-08T07:50:21.899-0400 7fdea52ae340 -1 librbd::Operations: cannot update immutable features (22) Invalid argument Any help or input is great appreciated. This message and any attachments may contain information that is protected by law as privileged and confidential, and is transmitted for the sole use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any use, dissemination, copying or retention of this e-mail or the information contained herein is strictly prohibited. If you received this e-mail in error, please immediately notify the sender by e-mail, and permanently delete this e-mail.

2 1

Removed host still active, sort of?
by Jeremy Hansen 11 Jun '23

11 Jun '23

I’m going through the process of transitioning to new hardware. Pacific 16.2.11. I drained the host, all daemons were removed. Did the ceph orch host rm <hostname> [ceph: root@cn01 /]# ceph orch host rm cn03.ceph Error EINVAL: host cn03.ceph does not exist Yet I see it here: ceph osd crush tree |grep cn03 -10 0 host cn03 Web interface says: ceph health HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm ceph orch host rm cn03.ceph --force Error EINVAL: host cn03.ceph does not exist No daemons are running. Something has a bad state. What can I do to clear this up? The previous host went without a problem and when all services were drained and I did the remove, it just completely disappeared as expected. Thanks -jeremy

1 3

Ceph drain not removing daemons
by Jeremy Hansen 09 Jun '23

09 Jun '23

I’m doing a drain on a host using cephadm, Pacific, 16.2.11. ceph orch host drain removed all the OSDs, but these daemons remain: grafana.cn06 cn06.ceph.la1 *:3000 stopped 5m ago 18M - - <unknown> <unknown> <unknown> mds.btc.cn06.euxhdu cn06.ceph.la1 running (2d) 5m ago 17M 29.4M - 16.2.11 de4b0b384ad4 017f7ef441ff mgr.cn06.rpkpwg cn06.ceph.la1 *:8443,9283 running (2d) 5m ago 10M 223M - 16.2.11 de4b0b384ad4 f1b89b453ef3 I manually stopped grafana. I expected these daemons to be removed as well. Is there an extra step I need to do here so I can remove the host cleanly? Thanks! -jeremy

1 1

Re: Disks are filling up
by Omar Siam 09 Jun '23

09 Jun '23

TL;DR: We could not fix this problem in the end and ended up with a Ceph fs in read only mode (so we could only backup, delete and restore) and one broken OSD (we deleted that and restored to a "new disk") I can now wrap up my whole experience with this problem. After the OSD usage growing to almost 2 TB x 3 OSDs (for a data that 'du' counted to be about 120GB), Ceph stopped filling up and in the week or two that followed, most of the space that was used showed up as free again. But there was one OSD that did not free up to meaningful values. To my surprise it was an OSD that is backed by SSDs not the one on HDD. It seems the biggest contributing factor was that I created the pools for the Ceph fs with autoscaling set to on (the default in cephadm dashboard). Now this pool never grew to more than 1 PG although it had a little over 100 GB on it. From what I read on this list this alone is prone to lock contention and other problematic behavior. Lessons learned: * If you use pool autoscaling, watch it if it actually does that. -> I opted for setting the new PG number for the replacement pools to 32 manually. * Ceph fs has a read only mode that at least lets you back up data in some bad states. -> That is good to know. It allows at least administrators to copy data to other storage devices. * If you use Ceph fs for persistent volumes in Kubernetes be aware that you probably loose all volumes at the same time when Ceph fs switches to read only. The CSI for Ceph fs does not work on read only Ceph fs, it always writes xattr on mount (the data pool that should be used and other internal data) and gives up if that fails. -> use a reasonable number of Ceph fs for Kubernetes persistent storage so you don't loose all PV at once. Other problems with my configuration I found: * We suffer from VMWare ESXi misreporting the type of disk physically attached. It reports some SAS SSDs as HDD. We also have real SAS HDD attached to some node. I suspected that to be a problem and we will exchange the HDDs for SSDs soon but the big problem was that the disktype was in the crushmap. -> Edited the crush map to ignore the type of storage as it is not very meaningful in our setup anyway * I had PGs stuck in undersized state for a long time and could not understand why ceph does not fix it. Then I checked the OSD weights (reweights) again and they were set to different values (1 and 0.85). After setting it to one on all OSDs Ceph actually started to bring all PG into the active+clean state. -> If all the OSDs actually are the same size i will either not reweight in the future or set it to the same value on all OSDs probably 1 So now after a noticable downtime of kubernetes and having to recreate most persistent volumes on the cluster Ceph health is HEALTH_OK again. I could upgrade to Ceph 16.2.13. I hope I can now upgrade to 17.2.6 without issues. Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at |www.oeaw.ac.at/acdh

1 0

ceph Pacific - MDS activity freezes when one the MDSs is restarted
by Emmanuel Jaep 09 Jun '23

09 Jun '23

Hi, we are currently running a ceph fs cluster at the following version: MDS version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) The cluster is composed of 7 active MDSs and 1 standby MDS: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 73 /s 1938k 1880k 85.3k 92.8k 1 active icadmin008 Reqs: 206 /s 2375k 2375k 7081 171k 2 active icadmin007 Reqs: 91 /s 5709k 5256k 149k 299k 3 active icadmin014 Reqs: 93 /s 679k 664k 40.1k 216k 4 active icadmin013 Reqs: 86 /s 3585k 3569k 12.7k 197k 5 active icadmin011 Reqs: 72 /s 225k 221k 8611 164k 6 active icadmin015 Reqs: 87 /s 1682k 1610k 27.9k 274k POOL TYPE USED AVAIL cephfs_metadata metadata 8552G 22.3T cephfs_data data 226T 22.3T STANDBY MDS icadmin006 When I restart one of the active MDSs, the standby MDS becomes active and its state becomes "replay". So far, so good! However, only one of the other "active" MDSs seems to remain active. All activities drop from the other ones: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 0 /s 1938k 1881k 85.3k 9720 1 active icadmin008 Reqs: 0 /s 2375k 2375k 7080 2505 2 active icadmin007 Reqs: 2 /s 5709k 5256k 149k 26.5k 3 active icadmin014 Reqs: 0 /s 679k 664k 40.1k 3259 4 replay icadmin006 801k 801k 1279 0 5 active icadmin011 Reqs: 0 /s 225k 221k 8611 9241 6 active icadmin015 Reqs: 0 /s 1682k 1610k 27.9k 34.8k POOL TYPE USED AVAIL cephfs_metadata metadata 8539G 22.8T cephfs_data data 225T 22.8T STANDBY MDS icadmin013 In effect, the cluster becomes almost unavailable until the newly promoted MDS finishes rejoining the cluster. Obviously, this defeats the purpose of having 7MDSs. Is this behavior? If not, what configuration items should I check to go back to "normal" operations? Best, Emmanuel

5 12

CreateMultipartUpload and Canned ACL - bucket-owner-full-control
by Rasool Almasi 09 Jun '23

09 Jun '23

A bucket with a policy that enforces "bucket-owner-full-control" results in Access Denied if multipart is used to upload the object. It is also discussed in an awscli issue: https://github.com/aws/aws-cli/issues/1674 aws client exits with "An error occurred (AccessDenied) when calling the CreateMultipartUpload operation", but it has been fixed on the server side rather than the client. Since without multi-part the object size in Rados Gateway has a limit of 5G and apparently increasing the limit is not recommended, how can I enforce "bucket-owner-full-control" and allow multipart upload? A sample bucket policy: { "Version": "2012-10-17", "Id": "read-write", "Statement": [ ... "Sid": "PolicyForAllowUploadWithCannedACL", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::18f2c0870e3649b0f383504aec1d0bef:root" ] }, "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::mybucket", "arn:aws:s3:::mybucket/*" ], "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } } } ] } Regards, Rasool

1 0

S3 and Omap
by xadhoom76＠gmail.com 09 Jun '23

09 Jun '23

Hi, we have a ceph 17.2.6 with ragosgw and a couple of buckets in it. We use it for backup with lock directly from veeam. After few backups we got HEALTH_WARN 2 large omap objects │····················· [WRN] LARGE_OMAP_OBJECTS: 2 large omap objects │····················· 2 large objects found in pool 'backup.rgw.buckets.index' │····················· Search the cluster log for 'Large omap object found' for more details. What is causing this ? could we get a bigger threshold avout omap size and ignore safely this warning ? What's the issue? Best regards

2 1

The pg_num from 1024 reduce to 32 spend much time, is there way to shorten the time？
by Louis Koo 09 Jun '23

09 Jun '23

ceph version is 16.2.13; The pg_num is 1024, and the target_pg_num is 32; there is no any data in the pool of ".rgw.buckets.index", but it spend much time to reduce the pg num.

5 12

rbd ls failed with operation not permitted
by zyz 08 Jun '23

08 Jun '23

When set the user's auth and then ls namespace, it is ok. But when I set the user's auth with namespace, ls namespace returns with error, but why?

2 1

[RGW] what is log_meta and log_data config in a multisite config?
by Gilles Mocellin 08 Jun '23

08 Jun '23

Hi Cephers, In a multisite config, with one zonegroup and 2 zones, when I look at `radiosgw-admin zonegroup get`, I see by defaut these two parameters : "log_meta": "false", "log_data": "true", Where can I find documentation on these, I can't find. I set log_meta to true, because, why not ? Is it a bad thing ?

2 2

Question about xattr and subvolumes
by Dario Graña 08 Jun '23

08 Jun '23

Hi, I'm installing a new instance (my first) of Ceph. Our cluster runs AlmaLinux9 + Quincy. Now I'm dealing with CephFS and quotas. I read documentation about setting up quotas with virtual attributes (xattr) and creating volumes and subvolumes with a prefixed size. I cannot distinguish which is the best option for us. Currently we create a directory with a project name and some subdirectories inside. I would like to understand the difference between both options. Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

keep rbd command history ever executed
by huxiaoyu＠horebdata.cn 08 Jun '23

08 Jun '23

Dear Ceph folks, In a Ceph cluster there could be multiple points (e.g. librbd clients) being able to execute rbd commands. My question is that , is there a methold to reliably record or keep a full rbd command history that ever being executed? This would be helpful for auditors as well as for system operators. any ideas? Samuel huxiaoyu(a)horebdata.cn

1 0

RadosGW S3 API Multi-Tenancy
by Brad House 08 Jun '23

08 Jun '23

I'm in the process of exploring if it is worthwhile to add RadosGW to our existing ceph cluster. We've had a few internal requests for exposing the S3 API for some of our business units, right now we just use the ceph cluster for VM disk image storage via RBD. Everything looks pretty straight forward until we hit multitenancy. The page on multi-tenancy doesn't dive into permission delegation: https://docs.ceph.com/en/quincy/radosgw/multitenancy/ The end goal I want is to be able to create a single user per tenant (Business Unit) which will act as their 'administrator', where they can then do basically whatever they want under their tenant sandbox (though I don't think we need more advanced cases like creations of roles or policies, just create/delete their own users, buckets, objects). I was hopeful this would just work, and I asked on the ceph IRC channel on OFTC and was told once I grant a user caps="users=*", they would then be allowed to create users *outside* of their own tenant using the Rados Admin API and that I should explore IAM roles. I think it would make sense to add a feature, such as a flag that can be set on a user, to ensure they stay in their "sandbox". I'd assume this is probably a common use-case. Anyhow, if its possible to do today using iam roles/policies, then great, unfortunately this is my first time looking at this stuff and there are some things not immediately obvious. I saw this online about AWS itself and creating a permissions boundary, but that's for allowing creation of roles within a boundary: https://www.qloudx.com/delegate-aws-iam-user-and-role-creation-without-givi… I'm not sure what "Action" is associated with the Rados Admin API create user for applying a boundary that the user can only create users with the same tenant name. https://docs.ceph.com/en/quincy/radosgw/adminops/#create-user Any guidance on this would be extremely helpful. Thanks! -Brad

1 1

16.2.13: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
by Zakhar Kirpichenko 08 Jun '23

08 Jun '23

Hi, I'm having an issue with crash daemons on Pacific 16.2.13 hosts. ceph-crash throws the following error on all hosts: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ceph-crash runs in docker, the container has the directory mounted: -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z The mount works correctly: 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 .. 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a 18:26 [root@ceph02 /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash [root@ceph02 /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. How can I convince it that the directory is there? Best regards, Zakhar

3 5

Bucket resharding in multisite without data replication
by Danny Webb 08 Jun '23

08 Jun '23

Hi Ceph users, We have 3 clusters running Pacific 16.2.9 all setup in a multisite configuration with no data replication (we wanted to use per bucket policies but never got them working to our satisfaction). All of the resharding documentation I've found regarding multisite is centred around multisite with data replication and having to reshard from the primary region and re-replicating data. But our data is spread amongst the regions and may not be in the primary region. Testing has shown that resharding from the primary region in the case of a bucket with data only in a remote region results in the remote bucket losing it's ability to list contents (seemingly breaking the index in the remote region). Is there a way (besides waiting for reef and dynamic bucket resharding for multisite) to reshard buckets in this setup? Cheers, Danny Danny Webb Principal OpenStack Engineer Danny.Webb(a)thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>

1 0

How to secure erasing a rbd image without encryption?
by huxiaoyu＠horebdata.cn 08 Jun '23

08 Jun '23

Dear ceph folks, I bumped into an very interesting challenge, how to secure erase a rbd image data without any encryption? The motivation is to ensure that there is no information leak on OSDs after deleting a user specified rbd image, without the extra burden of using rbd encryption. any ideas, suggestions are highly appreciated, Samuel huxiaoyu(a)horebdata.cn

4 3

Issues in installing old dumpling version to add a new monitor
by Cloud List 08 Jun '23

08 Jun '23

Hi, I have a very old Ceph cluster running the old dumpling version 0.67.1. One of the three monitors suffered a hardware failure and I am setting up a new server to replace the third monitor running Ubuntu 22.04 LTS (all the other monitors are using the old Ubuntu 12.04 LTS). I used ceph-deploy to deploy the cluster initially, and I can't use it since it's a very old version of ceph-deploy -- having issues with apt-key being deprecated and since ceph-deploy is no longer maintained, I can't upgrade it. And even if I can, I am not too sure if it works since the dumpling version is no longer in Ceph's official repository. So I tried to install it manually by cloning it from git: git clone -b dumpling https://github.com/ceph/ceph.git But when I tried to run "git submodule update --init" or "./autogen.sh" as per the README file, I am encountering this error: ==== root@ceph-mon-04:~/ceph-dumpling/ceph# git submodule update --init Submodule 'ceph-object-corpus' (git://ceph.com/git/ceph-object-corpus.git) registered for path 'ceph-object-corpus' Submodule 'src/libs3' (git://github.com/ceph/libs3.git) registered for path 'src/libs3' Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus'. Retry scheduled Cloning into '/root/ceph-dumpling/ceph/src/libs3'... Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus' a second time, aborting root@ceph-mon-04:~/ceph-dumpling/ceph# git submodule update --init --recursive Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus'. Retry scheduled Cloning into '/root/ceph-dumpling/ceph/ceph-object-corpus'... fatal: repository 'https://ceph.com/git/ceph-object-corpus.git/' not found fatal: clone of 'git://ceph.com/git/ceph-object-corpus.git' into submodule path '/root/ceph-dumpling/ceph/ceph-object-corpus' failed Failed to clone 'ceph-object-corpus' a second time, aborting root@ceph-mon-04:~/ceph-dumpling/ceph# ==== It seems that the repositories required for the submodules are no longer there. Anyone can advise me on the correct direction on how can I install the dumpling version of Ceph for me to add a new monitor? At the moment only 2 monitors out of 3 are up and I am worried that the cluster will be down if I lose another monitor. $ ceph status cluster 1660b11f-1074-4f5d-aa7c-64b479397a2f health HEALTH_WARN 1 mons down, quorum 0,1 ceph-mon-01,ceph-mon-02 What approach I should take: - Continue trying the manual installation/compiling route? - Continue trying the ceph-deploy route (by fixing the apt-key deprecation issue)? - Try to install the same old OS (Ubuntu 12.04 LTS) on the new server (not too sure if I still have the ISO) and see if it works? - Try to upgrade the current cluster and then add the monitor later after upgrade? (is it risky to upgrade with HEALTH_WARN status)? Any advice is greatly appreciated. Best regards, -ip-

3 2

slow mds requests with random read test
by Ben 07 Jun '23

07 Jun '23

Hi, We are performing couple performance tests on CephFS using fio. fio is run in k8s pod and 3 pods will be up running mounting the same pvc to CephFS volume. Here is command line for random read: fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=5 -runtime=500 -group_reporting -directory=/tmp/cache -name=Rand_Read_Testing_$BUILD_TIMESTAMP The random read is performed very slow. Here is the cluster log from dashboard: 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 33 secs 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:14 PM [WRN] Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:13 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:13:08 PM [WRN] slow request 34.213327 seconds old, received at 2023-05-30T12:12:33.951399+0000: client_request(client.270564:1406144 getattr pAsLsXsFs #0x700000103d0 2023-05-30T12:12:33.947323+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting 5/30/23 8:13:08 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 34.213328 secs 5/30/23 8:13:07 PM [WRN] slow request 33.169703 seconds old, received at 2023-05-30T12:12:33.952078+0000: peer_request:client.270564:1406144 currently dispatched 5/30/23 8:13:07 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 33.169704 secs 5/30/23 8:13:04 PM [INF] Cluster is now healthy 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 9 slow metadata IOs are blocked > 30 secs, oldest blocked for 45 secs 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 2 slow requests are blocked > 30 secs 5/30/23 8:12:57 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 44.954377 secs 5/30/23 8:12:52 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 39.954313 secs 5/30/23 8:12:48 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:12:47 PM [WRN] slow request 34.935921 seconds old, received at 2023-05-30T12:12:12.185614+0000: client_request(client.270564:1406139 create #0x7000001045b/atomic7966567911433736706tmp 2023-05-30T12:12:12.182999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] slow request 34.954254 seconds old, received at 2023-05-30T12:12:12.167281+0000: client_request(client.270564:1406138 rename #0x70000010457/build.xml #0x70000010457/atomic6590865221269854506tmp 2023-05-30T12:12:12.162999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] 2 slow requests, 2 included below; oldest blocked for > 34.954254 secs 5/30/23 8:12:44 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:12:41 PM [INF] Cluster is now healthy 5/30/23 8:12:41 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:12:41 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:12:40 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:12:40 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 38 secs However, random write test is performing very good. Any suggestions on the problem? Thanks, Ben

2 3

RGW: perf dump. What is "objecter-0x55b63c38fb80" ?
by Rudenko Aleksandr 07 Jun '23

07 Jun '23

Hi guys, In perf dump of RGW instance I have two similar sections. First one: "objecter": { "op_active": 0, "op_laggy": 0, "op_send": 38816, "op_send_bytes": 199927218, "op_resend": 0, "op_reply": 38816, "oplen_avg": { "avgcount": 38816, "sum": 90408 }, "op": 38816, "op_r": 12624, "op_w": 26192, "op_rmw": 0, "op_pg": 0, … } Second one: "objecter-0x55b63c38fb80": { "op_active": 0, "op_laggy": 0, "op_send": 5540, "op_send_bytes": 217343, "op_resend": 0, "op_reply": 5540, "oplen_avg": { "avgcount": 5540, "sum": 5636 }, "op": 5540, "op_r": 680, "op_w": 4860, "op_rmw": 0, "op_pg": 0, … } What is 0x55b63c38fb80 ? I try to monitor ‘op_active’ metric, but this metric refreshes only in ‘objecter-0x55b63c38fb80’ section and always 0 in ‘objecter’ section and it’s so difficult to monitor this metric because this id is dynamic and will be changed on next rgw restart.

1 0

Orchestration seems not to work
by Thomas Widhalm 07 Jun '23

07 Jun '23

Hi, I'm in the process of upgrading my cluster from 17.2.5 to 17.2.6 but the following problem existed when I was still everywhere on 17.2.5 . I had a major issue in my cluster which could be solved with a lot of your help and even more trial and error. Right now it seems that most is already fixed but I can't rule out that there's still some problem hidden. The very issue I'm asking about started during the repair. When I want to orchestrate the cluster, it logs the command but it doesn't do anything. No matter if I use ceph dashboard or "ceph orch" in "cephadm shell". I don't get any error message when I try to deploy new services, redeploy them etc. The log only says "scheduled" and that's it. Same when I change placement rules. Usually I use tags. But since they don't work anymore, too, I tried host and umanaged. No success. The only way I can actually start and stop containers is via systemctl from the host itself. When I run "ceph orch ls" or "ceph orch ps" I see services I deployed for testing being deleted (for weeks now). Ans especially a lot of old MDS are listed as "error" or "starting". The list doesn't match reality at all because I had to start them by hand. I tried "ceph mgr fail" and even a complete shutdown of the whole cluster with all nodes including all mgs, mds even osd - everything during a maintenance window. Didn't change anything. Could you help me? To be honest I'm still rather new to Ceph and since I didn't find anything in the logs that caught my eye I would be thankful for hints how to debug. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

3 19

Encryption per user Howto
by huxiaoyu＠horebdata.cn 07 Jun '23

07 Jun '23

Dear Ceph folks, Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store. Does anyone know (or have experience) how to do with CephFS and Ceph RGW? Any suggestionns or comments are highly appreciated, best regards, Samuel huxiaoyu(a)horebdata.cn

8 27

change user root to non-root after deploy cluster by cephadm
by farhad kh 07 Jun '23

07 Jun '23

Hi guys I deployed the ceph cluster with cephadm and root user, but I need to change the user to a non-root user And I did these steps: 1- Created a non-root user on all hosts with access without password and sudo `$USER_NAME ALL = (root) NOPASSWD:ALL` 2- Generated a SSH key pair and use ssh-copy-it to add all hosts ` ssh-keygen (accept the default file name and leave the passphrase empty) ssh-copy-id USER_NAME@HOST_NAME ` 3 - ceph cephadm set-user <user>But I get "Error EINVAL: ssh connection to root@hostname failed" error How to deal with this issue? What should be done to change the user to non-root?

2 1

Non cephadm cluster upgrade from octopus to quincy
by Szabo, Istvan (Agoda) 07 Jun '23

07 Jun '23

Hi, I don't find any documentation for this upgrade process. Is there anybody who has already done it yet? Is the normal apt-get update method works? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

reef v18.1.0 QE Validation status
by Yuri Weinstein 07 Jun '23

07 Jun '23

Details of this release are summarized here: https://tracker.ceph.com/issues/61515#note-1 Release Notes - TBD Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to merge https://github.com/ceph/ceph/pull/51788 for the core) rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - deprecated upgrade/pacific-x - known issues, Ilya, Laura? upgrade/reef-p2p - N/A clients upgrades - not run yet powercycle - Brad ceph-volume - in progress Please reply to this email with approval and/or trackers of known issues/PRs to address them. gibba upgrade was done and will need to be done again this week. LRC upgrade TBD TIA

7 9

Re: Workload Separation in Ceph RGW Cluster - Recommended or Not?
by Ramin Najjarbashi 06 Jun '23

06 Jun '23

Thank you for your response and for raising an important question regarding the potential bottlenecks within the RGW or the overall Ceph cluster. I appreciate your insight and would like to provide more information about the issues I have been experiencing. In my deployment, RGW instances 17-20 have been encountering problems such as hanging or returning errors, including "failed to read header: The socket was closed due to a timeout" and "res_query() failed." These issues have led to disruptions and congestions within the cluster. The index pool is indeed placed on a large number of NVMe SSDs to ensure fast access and efficient indexing of data. The number of Placement Groups (PGs) allocated for the index pool is also configured to be sufficient for the workload On Tue, Jun 6, 2023 at 21:27 Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Do you have reason to believe that your bottlenecks are within RGW not > within the cluster? > > e.g. is your index pool on a large number of NVMe SSDs with sufficient > PGs? Is your bucket data on SSD as well? > > > On Jun 6, 2023, at 13:52, Ramin Najjarbashi <ramin.najarbashi(a)gmail.com> > wrote: > > I would like to seek your insights and recommendations regarding the > practice of workload separation in a Ceph RGW (RADOS Gateway) cluster. I > have been facing challenges with large queues in my deployment and would > appreciate your expertise in determining whether workload separation is a > recommended approach or not. > > >

1 0

Workload Separation in Ceph RGW Cluster - Recommended or Not?
by Ramin Najjarbashi 06 Jun '23

06 Jun '23

Hi I would like to seek your insights and recommendations regarding the practice of workload separation in a Ceph RGW (RADOS Gateway) cluster. I have been facing challenges with large queues in my deployment and would appreciate your expertise in determining whether workload separation is a recommended approach or not. In my current Ceph cluster, I have 20 RGW instances. Client requests are directed to RGW1-16, while RGW17-20 are dedicated to administrative tasks and backend usage. However, I have been encountering errors and congestion issues due to the accumulation of large queues within the RGW instances. Considering the above scenario, I would like to inquire about your opinions on workload separation as a potential solution. Specifically, I am interested in knowing whether workload separation is recommended in a Ceph RGW cluster. To address the queue congestion and improve performance, my proposed solution includes separating the RGW instances based on their specific purposes. This entails allocating dedicated instances for client requests, backend usage, administrative tasks, metadata synchronization with other zone groups, garbage collection (GC), and lifecycle (LC) operations. I kindly request your feedback and insights on the following points: 1. Is workload separation considered a recommended practice in Ceph RGW deployments? 2. What are the potential benefits and drawbacks of workload separation in terms of performance, resource utilization, and manageability? 3. Are there any specific considerations or best practices to keep in mind while implementing workload separation in a Ceph RGW cluster? 4. Can you share your experiences or any references/documentation that highlight successful implementations of workload separation in Ceph RGW deployments? I truly value your expertise and appreciate your time and effort in providing guidance on this matter. Your insights will contribute significantly to optimizing the performance and stability of my Ceph RGW cluster.

1 0

Quincy release -Swift integration with Keystone
by fsbiz＠yahoo.com 06 Jun '23

06 Jun '23

Hi folks, My ceph cluster with Quincy and Rocky9 is up and running. But I'm having issues with swift authenticating with keystone. Was wondering if I'm missed anything in the configuration. From the debug logs below, it appears that radosgw is still trying to authenticate with Swift instead of Keystone. Any pointers will be appreciated. thanks, Here is my configuration. # ceph config dump | grep rgw client advanced debug_rgw 20/20 client advanced rgw_keystone_accepted_roles admin,user * client advanced rgw_keystone_admin_domain Default * client advanced rgw_keystone_admin_password <secret> * client advanced rgw_keystone_admin_project service * client advanced rgw_keystone_admin_user ceph-ks-svc * client advanced rgw_keystone_api_version 3 client advanced rgw_keystone_implicit_tenants false * client advanced rgw_keystone_token_cache_size 0 client basic rgw_keystone_url <Identity URL> * client advanced rgw_s3_auth_use_keystone true client advanced rgw_swift_account_in_url true client basic rgw_thread_pool_size 512 client.rgw.s_rgw.dev-ipp1-u1-control01.ojmddc basic rgw_frontends beast port=7480 * client.rgw.s_rgw.dev-ipp1-u1-control02.adnjrx basic rgw_frontends beast port=7480 Here's the debug log. If I interpret it correctly, it is trying to do a swift authentication and failing. Am I missing any configuration for Keystone based authentication ? Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: beast: 0x7fddeb8e7710: 10.117.53.10 - - [03/Jun/2023:18:47:03.060 +0000] "GET /swift/v1/AUTH_c668ed224e434c88a9e0fce125056112?format=json HTTP/1.1" 401 119 - "openstacksdk/0.52.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" - latency=0.000000000s Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_ACCEPT=*/* Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_ACCEPT_ENCODING=gzip, deflate Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_CONNECTION=close Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: HTTP_HOST=dev-ipp1-u1-object-store Jun 03 11:47:03 dev-ipp1-u1-control02radosgw[2802861]: HTTP_USER_AGENT=openstacksdk/0.52.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_VERSION=1.1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_X_AUTH_TOKEN=gAAAAABke4qn779UQ_XMz0EDL3P3TgjBQsGG6p-MNhviJxLZTuMTnTDmpT5Yfi9UpgO_T3LOOsPjQAw6zoMUIaC22wPeryp5x-UumB3XwXOWp-qSXLbuN3b9oj_Qg5kCZWA0waWNRHzQ1mwtlEmmpTgvTXbU5V1ym6hEBOn6Q3RWhn34Hj3cF9o Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_X_FORWARDED_FOR=10.117.148.3 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: QUERY_STRING=format=json Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REMOTE_ADDR=10.117.53.10 Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REQUEST_METHOD=GET Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REQUEST_URI=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112?format=json Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: SCRIPT_URI=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: SERVER_PORT=7480 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: ====== starting new request req=0x7fddeb8e7710 ===== Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s initializing for trans_id = tx000003991cfc5c1791f95-00647b8aa7-30c56-default Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s rgw api priority: s3=8 s3website=7 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s host=dev-ipp1-u1-object-store Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s name: format val: json Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s ver=v1 first= req= Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s handler=29RGWHandler_REST_Service_SWIFT Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s getting op 0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s get_system_obj_state: rctx=0x7fddeb8e6790 obj=default.rgw.log:script.prerequest. state=0x55f743b97720 s->prefetch_data=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets scheduling with throttler client=3 cost=1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets op=29RGWListBuckets_ObjStore_SWIFT Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets verifying requester Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::TempURLEngine denied with reason=-13 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::SignedTokenEngine denied with reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SwiftAnonymousEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::SwiftAnonymousEngine denied with reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets Failed the auth strategy, reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: failed to authorize request Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s op->ERRORHANDLER: err_no=-1 new_err_no=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s get_system_obj_state: rctx=0x7fddeb8e6790 obj=default.rgw.log:script.postrequest. state=0x55f743b97960 s->prefetch_data=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets op status=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets http status=401

2 1

RADOSGW integration with Keystone not working in Quincy release ??
by fsbiz＠yahoo.com 06 Jun '23

06 Jun '23

I have a ceph cluster installed using cephadm. The cluster is up and running but I'm unable to get Keystone integration working with RADOSGW. Is this a known issue? thanks,Fred.

1 0

Unexpected behavior of directory mtime after being set explicitly
by Sandip Divekar 06 Jun '23

06 Jun '23

Hi Team, I'm writing to bring to your attention an issue we have encountered with the "mtime" (modification time) behavior for directories in the Ceph filesystem. Upon observation, we have noticed that when the mtime of a directory (let's say: dir1) is explicitly changed in CephFS, subsequent additions of files or directories within 'dir1' fail to update the directory's mtime as expected. This behavior appears to be specific to CephFS - we have reproduced this issue on both Quincy and Pacific. Similar steps work as expected in the ext4 filesystem amongst others. Reproduction steps: 1. Create a directory - mkdir dir1 2. Modify mtime using the touch command - touch dir1 3. Create a file or directory inside of 'dir1' - mkdir dir1/dir2 Expected result: mtime for dir1 should change to the time the file or directory was created in step 3 Actual result: there was no change to the mtime for 'dir1' Note : For more detail, kindly find the attached logs. Our queries are : 1. Is this expected behavior for CephFS? 2. If so, can you explain why the directory behavior is inconsistent depending on whether the mtime for the directory has previously been manually updated. Best Regards, Sandip Divekar Component QA Lead SDET.

7 14

How to show used size of specific storage class in Radosgw?
by Huy Nguyen 06 Jun '23

06 Jun '23

Hi, I'm not able to find the information about used size of a storage class. - bucket stats - usage show - user stats ... Does Radosgw support it? Thanks

1 0

RGW: bucket notification issue with Kafka
by Huy Nguyen 06 Jun '23

06 Jun '23

Hi, In Ceph Radosgw 15.2.17, I get this issue when trying to create a push endpoint to Kafka Here is push endpoint configuration: endpoint_args = 'push-endpoint=kafka://abcef:123456@kafka.endpoint:9093&use-ssl=true&ca-location=/etc/ssl/certs/ca.crt' attributes = {nvp[0] : nvp[1] for nvp in urllib.parse.parse_qsl(endpoint_args, keep_blank_values=True)} response = snsclient.create_topic(Name=topic_name, Attributes=attributes) When I put an object, the radosgw log show this: Kafka connect: failed to create producer: ssl.ca.location failed: crypto/x509/by_file.c:199: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib: I have checked my ca.crt file and it is definitely in x509 format. If I use RGW v16.2.13, the producer will be created successfully. Anyone have any ideal? Thanks

1 0

RADOSGW not authenticating with Keystone. Quincy release
by fsbiz＠yahoo.com 05 Jun '23

05 Jun '23

Hi folks, My ceph cluster with Quincy and Rocky9 is up and running. But I'm having issues with RADOSGW authenticating with keystone. Was wondering if I'm missed anything in the configuration. From the debug logs below, it appears that radosgw is still trying to authenticate with Swift instead of Keystone. Any pointers will be appreciated. thanks, Here is my configuration. # ceph config dump | grep rgw client advanced debug_rgw 20/20 client advanced rgw_keystone_accepted_roles admin,user * client advanced rgw_keystone_admin_domain Default * client advanced rgw_keystone_admin_password <secret> * client advanced rgw_keystone_admin_project service * client advanced rgw_keystone_admin_user ceph-ks-svc * client advanced rgw_keystone_api_version 3 client advanced rgw_keystone_implicit_tenants false * client advanced rgw_keystone_token_cache_size 0 client basic rgw_keystone_url <Identity URL> * client advanced rgw_s3_auth_use_keystone true client advanced rgw_swift_account_in_url true client basic rgw_thread_pool_size 512 client.rgw.s_rgw.dev-ipp1-u1-control01.ojmddc basic rgw_frontends beast port=7480 * client.rgw.s_rgw.dev-ipp1-u1-control02.adnjrx basic rgw_frontends beast port=7480 Here's the debug log. If I interpret it correctly, it is trying to do a swift authentication and failing. Am I missing any configuration for Keystone based authentication ? Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: beast: 0x7fddeb8e7710: 10.117.53.10 - - [03/Jun/2023:18:47:03.060 +0000] "GET /swift/v1/AUTH_c668ed224e434c88a9e0fce125056112?format=json HTTP/1.1" 401 119 - "openstacksdk/0.52.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" - latency=0.000000000s Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_ACCEPT=*/* Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_ACCEPT_ENCODING=gzip, deflate Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_CONNECTION=close Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: HTTP_HOST=dev-ipp1-u1-object-store Jun 03 11:47:03 dev-ipp1-u1-control02radosgw[2802861]: HTTP_USER_AGENT=openstacksdk/0.52.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_VERSION=1.1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_X_AUTH_TOKEN=gAAAAABke4qn779UQ_XMz0EDL3P3TgjBQsGG6p-MNhviJxLZTuMTnTDmpT5Yfi9UpgO_T3LOOsPjQAw6zoMUIaC22wPeryp5x-UumB3XwXOWp-qSXLbuN3b9oj_Qg5kCZWA0waWNRHzQ1mwtlEmmpTgvTXbU5V1ym6hEBOn6Q3RWhn34Hj3cF9o Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_X_FORWARDED_FOR=10.117.148.3 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: QUERY_STRING=format=json Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REMOTE_ADDR=10.117.53.10 Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REQUEST_METHOD=GET Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REQUEST_URI=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112?format=json Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: SCRIPT_URI=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: SERVER_PORT=7480 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: ====== starting new request req=0x7fddeb8e7710 ===== Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s initializing for trans_id = tx000003991cfc5c1791f95-00647b8aa7-30c56-default Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s rgw api priority: s3=8 s3website=7 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s host=dev-ipp1-u1-object-store Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s name: format val: json Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s ver=v1 first= req= Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s handler=29RGWHandler_REST_Service_SWIFT Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s getting op 0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s get_system_obj_state: rctx=0x7fddeb8e6790 obj=default.rgw.log:script.prerequest. state=0x55f743b97720 s->prefetch_data=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets scheduling with throttler client=3 cost=1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets op=29RGWListBuckets_ObjStore_SWIFT Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets verifying requester Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::TempURLEngine denied with reason=-13 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::SignedTokenEngine denied with reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SwiftAnonymousEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::SwiftAnonymousEngine denied with reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets Failed the auth strategy, reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: failed to authorize request Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s op->ERRORHANDLER: err_no=-1 new_err_no=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s get_system_obj_state: rctx=0x7fddeb8e6790 obj=default.rgw.log:script.postrequest. state=0x55f743b97960 s->prefetch_data=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets op status=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets http status=401

1 0

PGs stuck undersized and not scrubbed
by Nicola Mori 05 Jun '23

05 Jun '23

Dear Ceph users, after an outage and recovery of one machine I have several PGs stuck in active+recovering+undersized+degraded+remapped. Furthermore, many PGs have not been (deep-)scrubbed in time. See below for status and health details. It's been like this for two days, with no recovery I/O being reported, so I guess something is stuck in a bad state. I'd need some help in understanding what's going on here and how to fix it. Thanks, Nicola --------------------- # ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 2 OSD(s) have spurious read errors Degraded data redundancy: 7349/147534197 objects degraded (0.005%), 22 pgs degraded, 22 pgs undersized 332 pgs not deep-scrubbed in time 503 pgs not scrubbed in time (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) services: mon: 5 daemons, quorum bofur,balin,aka,romolo,dwalin (age 2d) mgr: bofur.tklnrn(active, since 32h), standbys: balin.hvunfe, aka.wzystq mds: 2/2 daemons up, 1 standby osd: 104 osds: 104 up (since 37h), 104 in (since 37h); 22 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 18.53M objects, 40 TiB usage: 54 TiB used, 142 TiB / 196 TiB avail pgs: 7349/147534197 objects degraded (0.005%) 2715/147534197 objects misplaced (0.002%) 507 active+clean 20 active+recovering+undersized+degraded+remapped 2 active+recovery_wait+undersized+degraded+remapped # ceph health detail [WRN] PG_DEGRADED: Degraded data redundancy: 7349/147534197 objects degraded (0.005%), 22 pgs degraded, 22 pgs undersized pg 3.2c is stuck undersized for 37h, current state active+recovery_wait+undersized+degraded+remapped, last acting [79,83,34,37,65,NONE,18,95] pg 3.57 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,99,37,NONE,15,104,55,40] pg 3.76 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,5,37,15,100,33,85,NONE] pg 3.9c is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,86,88,NONE,11,69,20,10] pg 3.106 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,15,89,NONE,36,32,23,64] pg 3.107 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,NONE,64,20,61,92,104,43] pg 3.10c is stuck undersized for 37h, current state active+recovery_wait+undersized+degraded+remapped, last acting [79,34,NONE,95,104,16,69,18] pg 3.11e is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,89,64,46,32,NONE,40,15] pg 3.14e is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,34,69,97,85,NONE,46,62] pg 3.160 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,1,101,84,18,33,NONE,69] pg 3.16a is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,16,59,103,13,38,49,NONE] pg 3.16e is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [57,0,27,96,55,10,81,NONE] pg 3.170 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [NONE,57,14,46,55,99,15,40] pg 3.19b is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [NONE,79,59,8,32,17,7,90] pg 3.1a0 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [NONE,79,26,50,104,24,97,40] pg 3.1a5 is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,100,61,27,20,NONE,24,85] pg 3.1a8 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,24,NONE,3,55,40,98,45] pg 3.1aa is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,91,48,NONE,24,3,8,85] pg 3.1af is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,NONE,90,33,104,69,26,8] pg 3.1c1 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,95,NONE,53,54,27,18,85] pg 3.1c4 is stuck undersized for 2d, current state active+recovering+undersized+degraded+remapped, last acting [79,69,56,84,95,8,NONE,4] pg 3.1d5 is stuck undersized for 37h, current state active+recovering+undersized+degraded+remapped, last acting [57,48,NONE,104,34,16,37,89] [WRN] PG_NOT_DEEP_SCRUBBED: 332 pgs not deep-scrubbed in time pg 3.1ff not deep-scrubbed since 2023-05-18T21:06:57.883787+0000 pg 3.1fe not deep-scrubbed since 2023-05-22T19:50:11.497538+0000 pg 3.1fd not deep-scrubbed since 2023-05-22T19:44:12.680598+0000 pg 3.1fc not deep-scrubbed since 2023-05-20T19:56:43.746580+0000 pg 3.1fb not deep-scrubbed since 2023-05-22T18:29:12.794152+0000 pg 3.1f9 not deep-scrubbed since 2023-05-19T08:19:16.636964+0000 pg 3.1f8 not deep-scrubbed since 2023-05-22T21:49:28.891350+0000 pg 3.1f5 not deep-scrubbed since 2023-05-18T21:18:19.636068+0000 pg 3.1f4 not deep-scrubbed since 2023-05-18T18:00:41.241562+0000 pg 3.1f3 not deep-scrubbed since 2023-05-21T01:36:32.735139+0000 pg 3.1f2 not deep-scrubbed since 2023-05-23T03:59:02.154966+0000 pg 3.1f1 not deep-scrubbed since 2023-05-22T21:47:46.419880+0000 pg 3.1f0 not deep-scrubbed since 2023-05-22T19:17:38.327356+0000 pg 3.1ef not deep-scrubbed since 2023-05-19T01:49:04.133392+0000 pg 3.1ee not deep-scrubbed since 2023-05-21T12:25:52.010406+0000 pg 3.1ed not deep-scrubbed since 2023-05-19T20:13:20.675257+0000 pg 3.1eb not deep-scrubbed since 2023-05-18T12:13:53.684650+0000 pg 3.1ea not deep-scrubbed since 2023-05-18T09:45:57.172578+0000 pg 3.1e9 not deep-scrubbed since 2023-05-23T00:26:18.621324+0000 pg 3.1e8 not deep-scrubbed since 2023-05-21T05:15:03.969687+0000 pg 3.1e4 not deep-scrubbed since 2023-05-21T16:21:11.738145+0000 pg 3.1e3 not deep-scrubbed since 2023-05-22T13:13:19.611165+0000 pg 3.1e0 not deep-scrubbed since 2023-05-21T17:43:36.545240+0000 pg 3.1de not deep-scrubbed since 2023-05-18T00:03:49.873073+0000 pg 3.1dd not deep-scrubbed since 2023-05-22T20:30:56.025015+0000 pg 3.1db not deep-scrubbed since 2023-05-22T18:12:44.615539+0000 pg 3.1da not deep-scrubbed since 2023-05-20T21:11:00.060022+0000 pg 3.1d9 not deep-scrubbed since 2023-05-22T19:02:03.292022+0000 pg 3.1d8 not deep-scrubbed since 2023-05-23T17:37:05.320161+0000 pg 3.1d6 not deep-scrubbed since 2023-05-19T15:19:58.293551+0000 pg 3.1d4 not deep-scrubbed since 2023-05-23T02:28:54.392188+0000 pg 3.1d3 not deep-scrubbed since 2023-05-18T06:02:14.181321+0000 pg 3.1d2 not deep-scrubbed since 2023-05-18T11:46:29.582700+0000 pg 3.1d1 not deep-scrubbed since 2023-05-19T08:31:54.033426+0000 pg 3.1cd not deep-scrubbed since 2023-05-21T08:52:41.817826+0000 pg 3.1cc not deep-scrubbed since 2023-05-22T22:51:02.466708+0000 pg 3.1c9 not deep-scrubbed since 2023-05-18T08:06:50.220587+0000 pg 3.1c7 not deep-scrubbed since 2023-05-22T17:07:35.346608+0000 pg 3.1c5 not deep-scrubbed since 2023-05-20T17:09:12.048012+0000 pg 3.1c1 not deep-scrubbed since 2023-05-21T11:39:47.640196+0000 pg 3.1c0 not deep-scrubbed since 2023-05-22T20:22:57.166475+0000 pg 3.1bf not deep-scrubbed since 2023-05-19T19:08:08.313143+0000 pg 3.1be not deep-scrubbed since 2023-05-21T12:28:17.345386+0000 pg 3.1bd not deep-scrubbed since 2023-05-18T19:19:29.002801+0000 pg 3.1bb not deep-scrubbed since 2023-05-19T07:15:53.508751+0000 pg 3.1b8 not deep-scrubbed since 2023-05-19T18:50:27.701909+0000 pg 3.1b6 not deep-scrubbed since 2023-05-19T03:30:55.707248+0000 pg 3.1b5 not deep-scrubbed since 2023-05-20T20:37:48.346272+0000 pg 3.1b4 not deep-scrubbed since 2023-05-23T02:11:04.833784+0000 pg 3.1b3 not deep-scrubbed since 2023-05-18T20:46:40.876590+0000 282 more pgs... [WRN] PG_NOT_SCRUBBED: 503 pgs not scrubbed in time pg 3.1ff not scrubbed since 2023-05-24T23:37:22.323516+0000 pg 3.1fe not scrubbed since 2023-05-25T02:01:18.754476+0000 pg 3.1fd not scrubbed since 2023-05-24T20:31:23.239794+0000 pg 3.1fc not scrubbed since 2023-05-25T00:42:05.670791+0000 pg 3.1fb not scrubbed since 2023-05-24T19:29:29.438626+0000 pg 3.1fa not scrubbed since 2023-05-24T21:50:04.911965+0000 pg 3.1f9 not scrubbed since 2023-05-25T20:44:49.010622+0000 pg 3.1f8 not scrubbed since 2023-05-24T18:17:49.471926+0000 pg 3.1f7 not scrubbed since 2023-05-24T17:27:43.545337+0000 pg 3.1f6 not scrubbed since 2023-05-24T22:16:04.008644+0000 pg 3.1f5 not scrubbed since 2023-05-24T20:14:01.159271+0000 pg 3.1f4 not scrubbed since 2023-05-24T16:20:29.746958+0000 pg 3.1f3 not scrubbed since 2023-05-25T00:45:49.464448+0000 pg 3.1f2 not scrubbed since 2023-05-24T17:37:58.701570+0000 pg 3.1f1 not scrubbed since 2023-05-24T20:21:46.824657+0000 pg 3.1f0 not scrubbed since 2023-05-25T00:59:02.693836+0000 pg 3.1ef not scrubbed since 2023-05-24T21:35:10.061965+0000 pg 3.1ee not scrubbed since 2023-05-24T17:13:37.835095+0000 pg 3.1ed not scrubbed since 2023-05-24T18:17:21.739348+0000 pg 3.1ec not scrubbed since 2023-05-24T17:54:23.365899+0000 pg 3.1eb not scrubbed since 2023-05-24T23:18:31.345229+0000 pg 3.1ea not scrubbed since 2023-05-25T00:25:06.747723+0000 pg 3.1e9 not scrubbed since 2023-05-25T19:27:39.496774+0000 pg 3.1e8 not scrubbed since 2023-05-25T01:31:11.083814+0000 pg 3.1e7 not scrubbed since 2023-05-25T01:43:43.116599+0000 pg 3.1e6 not scrubbed since 2023-05-24T18:26:39.778008+0000 pg 3.1e4 not scrubbed since 2023-05-24T22:18:59.986309+0000 pg 3.1e3 not scrubbed since 2023-05-24T14:34:52.095564+0000 pg 3.1e2 not scrubbed since 2023-05-24T23:56:04.083842+0000 pg 3.1e1 not scrubbed since 2023-05-25T02:00:18.766811+0000 pg 3.1e0 not scrubbed since 2023-05-25T02:01:42.094304+0000 pg 3.1df not scrubbed since 2023-05-24T19:41:59.890557+0000 pg 3.1de not scrubbed since 2023-05-24T23:57:49.463552+0000 pg 3.1dd not scrubbed since 2023-05-25T17:42:33.397660+0000 pg 3.1dc not scrubbed since 2023-05-24T17:34:43.656366+0000 pg 3.1db not scrubbed since 2023-05-24T21:48:10.126232+0000 pg 3.1da not scrubbed since 2023-05-24T17:54:43.136739+0000 pg 3.1d9 not scrubbed since 2023-05-24T20:22:14.256914+0000 pg 3.1d8 not scrubbed since 2023-05-24T23:34:56.555311+0000 pg 3.1d7 not scrubbed since 2023-05-25T18:08:08.689329+0000 pg 3.1d6 not scrubbed since 2023-05-24T20:23:30.301130+0000 pg 3.1d5 not scrubbed since 2023-05-25T20:30:25.691077+0000 pg 3.1d4 not scrubbed since 2023-05-24T21:21:46.923743+0000 pg 3.1d3 not scrubbed since 2023-05-24T18:12:50.468466+0000 pg 3.1d2 not scrubbed since 2023-05-24T20:33:32.376232+0000 pg 3.1d1 not scrubbed since 2023-05-24T20:32:55.981738+0000 pg 3.1d0 not scrubbed since 2023-05-24T18:16:51.195524+0000 pg 3.1cf not scrubbed since 2023-05-24T22:32:00.879058+0000 pg 3.1ce not scrubbed since 2023-05-25T02:46:02.834267+0000 pg 3.1cd not scrubbed since 2023-05-24T21:02:08.288116+0000 453 more pgs...

2 2

Duplicate help statements in Prometheus metrics in 16.2.13
by Andreas Haupt 05 Jun '23

05 Jun '23

Dear all, after the update to CEPH 16.2.13 the Prometheus exporter is wrongly exporting multiple metric help & type lines for ceph_pg_objects_repaired: [mon1] /root #curl -sS http://localhost:9283/metrics # HELP ceph_pg_objects_repaired Number of objects repaired in a pool Count # TYPE ceph_pg_objects_repaired counter ceph_pg_objects_repaired{poolid="34"} 0.0 # HELP ceph_pg_objects_repaired Number of objects repaired in a pool Count # TYPE ceph_pg_objects_repaired counter ceph_pg_objects_repaired{poolid="33"} 0.0 # HELP ceph_pg_objects_repaired Number of objects repaired in a pool Count # TYPE ceph_pg_objects_repaired counter ceph_pg_objects_repaired{poolid="32"} 0.0 [...] This annoys our exporter_exporter service so it rejects the export of ceph metrics. Is this a known issue? Will this be fixed in the next update? Cheers, Andreas -- | Andreas Haupt | E-Mail: andreas.haupt(a)desy.de | DESY Zeuthen | WWW: http://www.zeuthen.desy.de/~ahaupt | Platanenallee 6 | Phone: +49/33762/7-7359 | D-15738 Zeuthen | Fax: +49/33762/7-7216

2 1

Converting to cephadm : Error EINVAL: Failed to connect
by David Barton 03 Jun '23

03 Jun '23

I am trying to debug an issue with ceph orch host add Is there a way to debug the specific ssh commands being issued or add debugging code to a python script? There is nothing useful in my syslog or /var/log/ceph/cephadm.log Is there a way to get the command to log, or can someone point me in the direction of the source code so I can have a look? I've run tcpdump on port 22 to listen for outgoing packets and also for traffic going to the target IP, and there is nothing going out when I run ceph orch host add If I run ssh inside the cephadm shell then I see the packets go out and it works as I document below. I was going to upgrade to Quincy from Pacific 16.2.5 and decided to upgrade from ceph-deploy to cephadm I initially had problems because I run ssh on a non-standard port. Allowing port 22 has allowed me to run the command below on every node except one. ceph orch host add [short hostname] [ip address] That one host fails, inexplicably with the error: Error EINVAL: Failed to connect to cephstorage-rs01 (103.XXX.YY.ZZ). If I run cephadm shell (without --no-hosts as that gives the error: unknown flag: --no-hosts) it works as expected. # cephadm shell Inferring fsid 525ec8aa-b401-4ddf-aa8f-4493727dac02 Inferring config /var/lib/ceph/525ec8aa-b401-4ddf-aa8f-4493727dac02/mon.cephstorage-ig03/config Using recent ceph image ceph/daemon-base@sha256:a038c6dc35064edff40bb7e824783f1bbd325c888e722ec5e814671406216ad5 root@cephstorage-ig03:/# ceph cephadm get-ssh-config > ssh_config root@cephstorage-ig03:/# ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key root@cephstorage-ig03:/# chmod 0600 ~/cephadm_private_key root@cephstorage-ig03:/# ssh -F ssh_config -i ~/cephadm_private_key root(a)103.XXX.YY.ZZ Warning: Permanently added '103.XXX.YY.ZZ' (ECDSA) to the list of known hosts. Welcome to XXXXX I was mucking around with custom ssh-config files to get around the port issue, but it did not seem to work so I and reverted back to the vanilla version with: ceph cephadm clear-ssh-config So when I am inside the shell it works, but it doesn't work properly via ceph orch host add There is one thing that is unusual that I think is worth mentioning. When I was adding the servers with custom ssh config files, I had a bad entry in the hosts file for cephstorage-rs01 on that server, resolving to 127.0.0.1 When I added it, it said it added the IP as 127.0.0.127# ceph orch host ls HOST ADDR LABELS STATUS ... cephstorage-rs01 127.0.0.127 Offline ... I then ran ceph orch host rm cephstorage-rs01 I have tried an iptables re-route in the vain idea that if there was some kind of host to IP cache it would route to localhost and tell me that the host name didn't match. That did not work. Right now, I am a sad panda as my ceph cluster is half transitioned. My next port of call is probably to try and adopt what I can into cephadm and make sure the cluster is ok, and then finally drop the problem node and then re-add it. Any help will be appreciated. Regards, David

3 3

Re: CEPH Version choice
by Frank Schilder 02 Jun '23

02 Jun '23

Hi Marc, I uploaded all scripts and a rudimentary readme to https://github.com/frans42/cephfs-bench . I hope it is sufficient to get started. I'm afraid its very much tailored to our deployment and I can't make it fully configurable anytime soon. I hope it serves a purpose though - at least I discovered a few bugs with it. We actually kept the benchmark running through an upgrade from mimic to octopus. Was quite interesting to see how certain performance properties change with that. This benchmark makes it possible to compare versions with live timings coming in. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Marc <Marc(a)f1-outsourcing.eu> Sent: Monday, May 15, 2023 11:28 PM To: Frank Schilder Subject: RE: [ceph-users] Re: CEPH Version choice > I planned to put it on-line. The hold-back is that the main test is un- > taring a nasty archive and this archive might contain personal > information, so I can't just upload it as is. I can try to put together > a similar archive from public sources. Please give me a bit of time. I'm > also a bit under stress right now with our users being hit by an FS meta > data corruption. That's also why I'm a bit trigger happy. > Ok thanks, very nice, no hurry!!!

2 2

NFS export of 2 disjoint sub-dir mounts
by Frank Schilder 02 Jun '23

02 Jun '23

Hi all, I have a problem with exporting 2 different sub-folder ceph-fs kernel mounts via nfsd to the same IP address. The top-level structure on the ceph fs is something like /A/S1 and /A/S2. On a file server I mount /A/S1 and /A/S2 as two different file systems under /mnt/S1 and /mnt/S2 using the ceph fs kernel client. Then, these 2 mounts are exported with lines like these in /etc/exports: /mnt/S1 -options NET /mnt/S2 -options IP IP is an element of NET, meaning that the host at IP should be the only host being able to access /mnt/S1 and /mnt/S2. What we observe is that any attempt to mount the export /mnt/S1 on the host at IP results in /mnt/S2 being mounted instead. My first guess was that here we have a clash of fsids and the ceph fs is simply reporting the same fsid to nfsd and, hence, nfsd thinks both mountpoints contain the same. So I modified the second export line to /mnt/S2 -options,fsid=100 IP to no avail. The two folders are completely disjoint, neither symlinks nor hard-links between them. So it should be safe to export these as 2 different file systems. Exporting such constructs to non-overlapping networks/IPs works as expected - even when exporting subdirs of a dir (like exporting /A/B and /A/B/C from the same file server to strictly different IPs). It seems the same-IP config that breaks expectations. Am I missing here a magic -yes-i-really-know-what-i-am-doing hack? The file server is on AlmaLinux release 8.7 (Stone Smilodon) and all ceph packages match the ceph version octopus latest of our cluster. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

Metadata pool space usage decreases
by Nathan MALO 02 Jun '23

02 Jun '23

Hello all, I have a weird behavior on my Cephfs. On May 29th I noticed a drop of 50Tb in my data pool. It has been followed by a decrease of space usage in the metadata pool since then. From May 29th, still happening as I write, the metadata pool has lost 1Tb over the initial 1.8Tb. Regarding the number of objects, it was 8.4 Million and is now 7.8 Million. I assume that my users have deleted a lot a of files on that day (our cephfs consists a very small files of about 4Mb). But such a huge decrease in metadata pool got me really concerned, it really seems huge. I thought that maybe it was the mds lazy deletion happening but I am not sure. Does anyone have any thought about this ? Do you know more about lazy deletion ? There is not much documentation about this online. Do you know any way, command, logfile to search in order to get the current lazy deletion operations ? I noticed that the read_ops, read_bytes, write_ops and write_bytes reported by the command rados df detail are negative for the metadata pool. My cluster is running Nautilus. Any help would be appreciated, Best regards, Nate

1 0

PGs incomplete - Data loss
by Benno Wulf 01 Jun '23

01 Jun '23

Hi guys, I'm awake since 36h and try to restore a broken ceph Pool (2 PGs incomplete) My vm are all broken. Some Boot, some Dont Boot... Also I have 5 removed disk with Data of that Pool "in my Hands" - Dont ask... So my question is it possible to restore Data of these other disks and "add" them thee others for healing? Best regards Ben

2 1

Cluster without messenger v1, new MON still binds to port 6789
by Robert Sander 01 Jun '23

01 Jun '23

Hi, a cluster has ms_bind_msgr1 set to false in the config database. Newly created MONs still listen on port 6789 and add themselves as providing messenger v1 into the monmap. How do I change that? Shouldn't the MONs use the config for ms_bind_msgr1? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 1

How to disable S3 ACL in radosgw
by Rasool Almasi 01 Jun '23

01 Jun '23

Hi, Is it possible to disable ACL in favor of bucket policy (on a bucket or globally)? The goal is to forbid users to use any bucket/object ACLs and only allow bucket policies. Seems there is no documentation in that regard which applies to Ceph RGW. Apology if I am sending this in the wrong mailing list. Regards, Rasool

1 0

cephadm does not honor container_image default value
by Daniel Krambrock 01 Jun '23

01 Jun '23

Hello. I think i found a bug in cephadm/ceph orch: Redeploying a container image (tested with alertmanager) after removing a custom `mgr/cephadm/container_image_alertmanager` value, deploys the previous container image and not the default container image. I'm running `cephadm` from ubuntu 22.04 pkg 17.2.5-0ubuntu0.22.04.3 and `ceph` version 17.2.6. Here is an example. Node clrz20-08 is the node altermanager is running on, clrz20-01 the node I'm controlling ceph from: * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` * Set alertmanager image ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` * Remove alertmanager image setting, revert to default: ``` root@clrz20-01:~# ceph config rm mgr mgr/cephadm/container_image_alertmanager root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:latest" ``` -> `mgr/cephadm/container_image_alertmanager` is set to `quay.io/prometheus/alertmanager:v0.23.0`, but redeploy uses `quay.io/prometheus/alertmanager:latest`. This looks like a bug. * Set alertmanager image explicitly to the default value ``` root@clrz20-01:~# ceph config set mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 root@clrz20-01:~# ceph config get mgr mgr/cephadm/container_image_alertmanager quay.io/prometheus/alertmanager:v0.23.0 ``` * redeploy altermanager ``` root@clrz20-01:~# ceph orch redeploy alertmanager Scheduled to redeploy alertmanager.clrz20-08 on host 'clrz20-08' ``` * Get alertmanager version ``` root@clrz20-08:~# cephadm ls | jq '.[] | select(.service_name == "alertmanager")| .container_image_name' "quay.io/prometheus/alertmanager:v0.23.0" ``` -> Setting `mgr/cephadm/container_image_alertmanager` to the default setting fixes the issue. Bests, Daniel

3 3

How to specify to only build ceph-radosgw package from source?
by huy nguyen 01 Jun '23

01 Jun '23

Hi, I usually install the SRPM and then build from ceph.spec like this: rpmbuild -bb /root/rpmbuild/SPECS/ceph.spec --without ceph_test_package But it take a long time and contain many packages that I don't need. So is there a way to optimize this build process for only needed package, for example ceph-radosgw? Thanks.

1 0