- ceph-users - lists.ceph.io

OSD bind ports min/max sizing
by Szabo, Istvan (Agoda) 16 Dec '24

16 Dec '24

Hi, Going through this documentation: https://docs.redhat.com/en/documentation/red_hat_ceph_storage/2/html/config… Wonder the 6800-7100 range consist how many osd normally? How do you size this in ceph? Thank you

3 2

failed to load OSD map for epoch 2898146, got 0 bytes
by Frank Schilder 15 Dec '24

15 Dec '24

Hi all, I have a strange problem on an octopus latest cluster. We had a couple of SSD OSDs down for a while and brought them up today again. For some reason, these OSDs don't come up and flood the log with messages like osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages cycle through the same epochs over and over again. I did not really fine too much help, there is an old thread about a similar/the same problem on a home lab cluster, with new OSDs though, I believe. I couldn't really find useful information. The OSDs seem to boot fine and then end up in something like a death loop. Below some snippets from the OSD log. Any hints appreciated. Thanks and best regards, Frank After OSD start, everything looks normal up to here: 2024-10-21T17:41:39.136+0200 7fad73cf6f00 0 osd.1004 2971464 load_pgs opened 205 pgs 2024-10-21T17:41:39.140+0200 7fad73cf6f00 -1 osd.1004 2971464 log_to_monitors {default=true} 2024-10-21T17:41:39.150+0200 7fad73cf6f00 -1 osd.1004 2971464 mon_cmd_maybe_osd_create fail: 'osd.1004 has already bound to class 'fs_meta', can not reset class to 'ssd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy 2024-10-21T17:41:39.155+0200 7fad519a3700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad511a2700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4f99f700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898132, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898133, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898134, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898135, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898136, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898137, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad73cf6f00 0 osd.1004 2971464 done with init, starting boot process 2024-10-21T17:41:39.155+0200 7fad73cf6f00 1 osd.1004 2971464 start_boot 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898138, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898139, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898140, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898141, got 0 bytes 2024-10-21T17:41:39.155+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898142, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898143, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898144, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898145, got 0 bytes 2024-10-21T17:41:39.156+0200 7fad4b196700 -1 osd.1004 2971464 failed to load OSD map for epoch 2898146, got 0 bytes These messages repeat over and over again with some others of this form showing up every now and then: 2024-10-21T17:41:39.476+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82879: 76571 keys, 67866714 bytes 2024-10-21T17:41:39.688+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525299690000, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82879, "file_size": 67866714, "table_properties": {"data_size": 67111697, "index_size": 562601, "filter_size": 191557, "raw_key_size": 4823973, "raw_average_key_size": 63, "raw_value_size": 62631087, "raw_average_value_size": 817, "num_data_blocks": 15644, "num_entries": 76571, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} And another occasion: 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 12] Generated table #82880: 76774 keys, 67868330 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899234, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899235, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad501a0700 -1 osd.1004 2971464 failed to load OSD map for epoch 2899236, got 0 bytes 2024-10-21T17:41:40.520+0200 7fad651ca700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1729525300521403, "cf_name": "default", "job": 12, "event": "table_file_creation", "file_number": 82880, "file_size": 67868330, "table_properties": {"data_size": 67113021, "index_size": 562509, "filter_size": 191941, "raw_key_size": 4836742, "raw_average_key_size": 62, "raw_value_size": 62623274, "raw_average_value_size": 815, "num_data_blocks": 15630, "num_entries": 76774, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

6 12

OSD process in the "weird" state
by Jan Marek 14 Dec '24

14 Dec '24

Hello, we have sometimes problem with OSD process in the "weird" state - its "flapping" between dead and health state. CEPH cluster is in the version 18.2.2, installed by 'cephadm' bootstrap process. Cluster hosts uses for internal communication RoCE rdma: # ceph config dump: global advanced ms_cluster_type async+rdma * global advanced ms_public_type async+posix * Our netwok cards are: 43:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] 43:00.1 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] (2x100Gbps) We have all-flash NVMe configuration. When some OSD goes to the "half-dead" state, we have in our syslog this messages: 2024-12-10T00:56:55.025681+01:00 c1-dc1 ceph-osd[3346501]: RDMAStack handle_async_event QP not dead, event associate qp number: 80235 Queue Pair status: IBV_QPS_ERR Event : last WQE reached 2024-12-10T00:56:55.025777+01:00 c1-dc1 ceph-89ec5e54-dba6-11ee-9c3e-72791a95392b-osd-1[3346299]: 2024-12-09T23:56:55.018+0000 7f5b5950c700 -1 RDMAStack handle_async_event QP not dead, event associate qp number: 80235 Queue Pair status: IBV_QPS_ERR Event : last WQE reached We are using podman as a container solution. When we do restart this problematic container, its start and everythink goes smoothly. If we don't restart this container, in log we have a lot of messages, that some OSD maked this OSD as dead, but this OSD complaints about it and cluster goes to the problematic state. We are using this CEPH cluster as a storage for ProxMox virtualization and some virtuals don't "survive" this situation, as they have not accessible theire "disks" :-(. Is there a some solution, which we can a try? Many thanks for advices. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

2 1

[RGW] Never ending PUT requests
by Gilles Mocellin 13 Dec '24

13 Dec '24

Hello ! Since we have upgraded our Ceph cluster from Ceph Pacific 16.2.15 to Ceph 18.2.2. We can see more requests in timeout. Especially PUT from backup tools, Nexus Repository... Een with 10min timeouts on haproxy for the server backends. We are upgrading Ubuntu from 18.04 to 22.04 to be able to upgrade to Ceph 18.2.4, but not finished yet. With debug log level, I can see only one thing concerning a never ending request. Here are the last logs : Dec 09 12:50:20 server-xxx bash[876499]: debug 2024-12-09T11:50:20.189+0000 7f5af9890700 10 req 7017125235642647451 0.004000126s s3:put_obj setting object write_tag=3e3b0a2b-1864-4923-93a6-237a13b51594.81430478.7017125235642647451 Dec 09 12:50:20 server-xxx bash[876499]: debug 2024-12-09T11:50:20.189+0000 7f5af9890700 20 req 7017125235642647451 0.004000126s s3:put_obj bucket index object: MYREGION.rgw.buckets.index:.dir.3e3b0a2b-1864-4923-93a6-237a13b51594.80569777.1.5 Dec 09 12:50:20 server-xxx bash[876499]: debug 2024-12-09T11:50:20.341+0000 7f5a83fa5700 20 req 7017125235642647451 0.156004906s s3:put_obj WARNING: blocking librados call I wonder if that last line is the culprit, end if the prcess ends or stay blocked indefinitelly... --Gilles

1 0

Dashboard redirection changed after upgrade octopus to pacific
by Frank Schilder 13 Dec '24

13 Dec '24

Hi all, this here are 2 questions, one out of curiosity and one about performance when having dashboard access behind an nginx reverse proxy. We follow instructions here: https://docs.ceph.com/en/pacific/mgr/dashboard/#proxy-configuration. Question 2 about nginx: Is anyone using dashboard access via an nginx reverse proxy? My interpretation of the documentation is, that it will work (nginx handles the redirect response transparently) as long as the one MGR pointed to is up. However, I wonder how smart the handling of redirects is. Does nginx remember a redirected connection or will it go through the "ask configured host"->"receive redirect"->"ask redirection target" procedure for every single http(s) request, or will it remember the redirection for established connections? Question 1 out of curiosity: On octopus MGRs would return redirection URLs containing the FQDN of the host. Under pacific this does not work anymore. One can enable hostname resolution (https://docs.ceph.com/en/pacific/mgr/dashboard/#resolve-ip-address-to-hostn…) however, this requires a reverse DNS lookup and is off by default. How did octopus do that? Was it reverse DNS lookup and it was just on by default? If not, why is the simple "use result of hostname -f" that does not require DNS not an option? Thanks for answers to either or both of these questions! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

19.2.1 reediness for QE Validation
by Yuri Weinstein 13 Dec '24

13 Dec '24

All PRs in the 19.2.1 milestone have been merged. And I've run preliminary tests on "squid-release" branch.: https://pulpito.ceph.com/?sha1=a4b5f6b5b3e7cf446c6f6bafb60001513356af2c There are lots of errors 'DevicePath' in the different suites. Suspect is https://tracker.ceph.com/issues/69067 We will continue testing when this is addressed. Thx YuriW

3 4

Ceph Cluster slowness in production
by Michel Niyoyita 13 Dec '24

13 Dec '24

Hello team, I have configured a ceph cluster running on ubuntu 20.04 , with 16 HDD for data and 4 SSD for journaling on each host. the total cluster size is 150 TB usable with a replica of 3. The cluster is in production , integrated with openstack with different pools: Volumes, vms, cinder and cinder-backup. The problem I have is this : For linux servers there is no slowness , but on windows servers is worse , they are very very slow, some of those servers have been migrated from vmware to openstack using coriolis , the network interfaces and switches are 10Gbps. Kindly advise the way I can speed up my cluster to avoid the slowness. Best regards Michel

3 2

How to list pg-upmap-items
by Frank Schilder 13 Dec '24

13 Dec '24

Dear all, during our upgrade from octopus to pacific the MGR suddenly started logging messages like this one to audit.log: 2024-12-10T10:30:01.105524+0100 mon.ceph-03 (mon.2) 3004 : audit [INF] from='mgr.424622547 192.168.32.67:0/63' entity='mgr.ceph-03' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "1.60", "id": [1054, 1125]}]: dispatch Apparently, the balancer got enabled and tried to do something. However, we never enabled pg-upmap on our cluster, because we still have jewel clients from the museum connected. Therefore, I'm pretty certain that all of these upmap requests either failed or are scheduled and pending. To be sure, I would like to confirm that nothing happened. How can I list upmap items and scheduled+pending upmap operations? If there are any, how do I delete these? I really would like to avoid that these requests start hurting in a few years from now. I looked at the documentation. Unfortunately, its the usual disease[1], commands for setting all sorts of stuff are documented, but commands to query anything seem to be missing. This workaround [root@gnosis osdmaps]# ceph osd dump -f json-pretty | grep upmap "pg_upmap": [], "pg_upmap_items": [], indicates nothing is screwed up yet. However, I would really like to know what happened to the MGR commands and where they are now. How do I confirm they went to digital heaven? [1] There are "ceph osd pg-upmap-items : set upmap items" and "ceph osd rm-pg-upmap-items : clear upmap items" commands. Why would anyone ever need a "ceph osd ls-pg-upmap-items"?? I found out that I can write it myself (https://ceph-users.ceph.narkive.com/h7y24SDg/stale-pg-upmap-items-entries-a… and https://gitlab.cern.ch/ceph/ceph-scripts/blob/master/tools/upmap/upmap-rema…) However, a good API is always symmetric to make it *easy* for users to check and fix screw-ups. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 3

The Object Gateway Service is not configured, Credentials not found for RGW Daemon
by Malte Stroem 12 Dec '24

12 Dec '24

Hello, this issue hit lots of people in the past. Now, I am on 18.2.2 and after configuring multisite between two clusters the RGWs/Object Gateways are not visible in the dashboard anymore. There are two realms, two zonegroups now and for each realm three RGWs running on one cluster. One of the two clusters works fine and shows the RGWs in the dashboard. The other one shows: Message: The Object Gateway Service is not configured Credentials not found for RGW Daemon: '' Please consult the documentation on how to configure and enable the Object Gateway management functionality. Setting the credentials again did not solve the problem. Debugging the dashboard shows: mgr get_typed_config RGW_API_ADMIN_RESOURCE not found mgr get_typed_config ALERTMANAGER_API_SSL_VERIFY not found mgr get_typed_config PROMETHEUS_API_SSL_VERIFY not found However: ceph dashboard get-rgw-api-admin-resource admin and: ceph dashboard get-alertmanager-api-ssl-verify True and: ceph dashboard get-prometheus-api-ssl-verify True and: ceph dashboard get-rgw-api-ssl-verify False Some ideas for debugging deeper? Javascript console in the browser did not show anything. Best, Malte

2 2

NFS cluster
by Rok Jaklič 12 Dec '24

12 Dec '24

Hi, I am trying to create nfs cluster with following command: ceph nfs cluster create cephnfs But I get an error like: Error EPERM: osd pool create failed: 'pgp_num' must be greater than 0 and lower or equal than 'pg_num', which in this case is 1 retval: -34 Any ideas why? I also tried adding placement number at the end like: ceph nfs cluster create cephnfs 128 ... but I get the same error. Ceph version quincy: v17.2.8 Kind regards, Rok

2 1