- ceph-users - lists.ceph.io

How to trigger scrubbing in Ceph on-demand ?
by Jayjeet Chakraborty 26 Oct '23

26 Oct '23

Hi all, I am trying to trigger deep scrubbing in Ceph reef (18.2.0) on demand on a set of files that I randomly write to CephFS. I have tried both invoking deep-scrub on CephFS using ceph tell and just deep scrubbing a particular PG. Unfortunately, none of that seems to be working for me. I am monitoring the ceph status output, it never shows any scrubbing information. Can anyone please help me out on this ? In a nutshell, I need Ceph to scrub for me anytime I want. I am using Ceph with default configs for scrubbing. Thanks all. Best Regards, *Jayjeet Chakraborty* Ph.D. Student Department of Computer Science and Engineering University of California, Santa Cruz *Email: jayjeetc(a)ucsc.edu <jayjeetc(a)ucsc.edu>*

3 4

quincy v17.2.7 QE Validation status
by Yuri Weinstein 26 Oct '23

26 Oct '23

Details of this release are summarized here: https://tracker.ceph.com/issues/63219#note-2 Release Notes - TBD Issue https://tracker.ceph.com/issues/63192 appears to be failing several runs. Should it be fixed for this release? Seeking approvals/reviews for: smoke - Laura rados - Laura, Radek, Travis, Ernesto, Adam King rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/quincy-p2p - Known issue IIRC, Casey pls confirm/approve client-upgrade-quincy-reef - Laura powercycle - Brad pls confirm ceph-volume - Guillaume pls take a look Please reply to this email with approval and/or trackers of known issues/PRs to address them. Josh, Neha - gibba and LRC upgrades -- N/A for quincy now after reef release. Thx YuriW

10 25

[quincy - 17.2.6] Lua scripting in the rados gateway - HTTP_REMOTE-ADDR missing
by stephan＠gridscale.io 26 Oct '23

26 Oct '23

Hi Ceph users, currently I'm using the lua script feature in radosgw to send "put_obj" and "get_obj" requests stats to a mongo db. So far it's working quite well but I miss a field which is very important for us for traffic stats. Im looking for the HTTP_REMOTE-ADDR field which is available in the ops_log but couldn't find it in here https://docs.ceph.com/en/quincy/radosgw/lua-scripting/#request-fields Does someone know how to get this field via lua script? Cheers Stephan

2 2

Enterprise SSD require for Ceph Reef Cluster
by Nafiz Imtiaz 25 Oct '23

25 Oct '23

Hi, We have a ceph cluster running reef version. We want to buy some enterprise ssd for our ceph cluster, and our prepared storage size is 1.92TB. For that, we have selected the Intel model. Please give a review about this model, and if you have any other model preference, please share with us. Thank you Brand: Intel SSD: 1.92TB 2.5'' Enterprise SATA, 6Gb/s Model: D3-S4510 Regards, Nafiz Imtiaz Assistant Manager, Product Development IT Division Bangladesh Export Import Company Ltd.

1 0

init unable to update_crush_location: (34) Numerical result out of range
by Pardhiv Karri 25 Oct '23

25 Oct '23

Hi, Getting an error while adding a new node/OSD with bluestore OSDs to the cluster. The OSD is added without any host and is down, tried to bring it up didn't work. The same method to add in other clusters doesn't have any issue. Any idea what the problem is? Ceph Version: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable) Ceph Health: OK 2023-10-25 20:40:40.867878 7f1f478cde40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1698266440867866, "job": 1, "event": "recovery_started", "log_files": [270]} 2023-10-25 20:40:40.867883 7f1f478cde40 4 rocksdb: [/build/ceph-U0cfoi/ceph-12.2.11/src/rocksdb/db/db_impl_open.cc:482] Recovering log #270 mode 0 2023-10-25 20:40:40.867904 7f1f478cde40 4 rocksdb: [/build/ceph-U0cfoi/ceph-12.2.11/src/rocksdb/db/version_set.cc:2395] Creating manifest 272 2023-10-25 20:40:40.869553 7f1f478cde40 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1698266440869548, "job": 1, "event": "recovery_finished"} 2023-10-25 20:40:40.870924 7f1f478cde40 4 rocksdb: [/build/ceph-U0cfoi/ceph-12.2.11/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x55c9061ba000 2023-10-25 20:40:40.870964 7f1f478cde40 1 bluestore(/var/lib/ceph/osd/ceph-721) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152 2023-10-25 20:40:40.871234 7f1f478cde40 1 freelist init 2023-10-25 20:40:40.871293 7f1f478cde40 1 bluestore(/var/lib/ceph/osd/ceph-721) _open_alloc opening allocation metadata 2023-10-25 20:40:40.871314 7f1f478cde40 1 bluestore(/var/lib/ceph/osd/ceph-721) _open_alloc loaded 3.49TiB in 1 extents 2023-10-25 20:40:40.874700 7f1f478cde40 0 <cls> /build/ceph-U0cfoi/ceph-12.2.11/src/cls/cephfs/cls_cephfs.cc:197: loading cephfs 2023-10-25 20:40:40.874721 7f1f478cde40 0 _get_class not permitted to load sdk 2023-10-25 20:40:40.874955 7f1f478cde40 0 _get_class not permitted to load kvs 2023-10-25 20:40:40.875638 7f1f478cde40 0 _get_class not permitted to load lua 2023-10-25 20:40:40.875724 7f1f478cde40 0 <cls> /build/ceph-U0cfoi/ceph-12.2.11/src/cls/hello/cls_hello.cc:296: loading cls_hello 2023-10-25 20:40:40.875776 7f1f478cde40 0 osd.721 0 crush map has features 288232575208783872, adjusting msgr requires for clients 2023-10-25 20:40:40.875780 7f1f478cde40 0 osd.721 0 crush map has features 288232575208783872 was 8705, adjusting msgr requires for mons 2023-10-25 20:40:40.875784 7f1f478cde40 0 osd.721 0 crush map has features 288232575208783872, adjusting msgr requires for osds 2023-10-25 20:40:40.875837 7f1f478cde40 0 osd.721 0 load_pgs 2023-10-25 20:40:40.875840 7f1f478cde40 0 osd.721 0 load_pgs opened 0 pgs 2023-10-25 20:40:40.875844 7f1f478cde40 0 osd.721 0 using weightedpriority op queue with priority op cut off at 64. 2023-10-25 20:40:40.877401 7f1f478cde40 -1 osd.721 0 log_to_monitors {default=true} 2023-10-25 20:40:40.888408 7f1f478cde40 -1 osd.721 0 mon_cmd_maybe_osd_create fail: '(34) Numerical result out of range': (34) Numerical result out of range 2023-10-25 20:40:40.891367 7f1f478cde40 -1 osd.721 0 mon_cmd_maybe_osd_create fail: '(34) Numerical result out of range': (34) Numerical result out of range 2023-10-25 20:40:40.891409 7f1f478cde40 -1 osd.721 0 init unable to update_crush_location: (34) Numerical result out of range Thanks, Pardhiv

1 0

cephadm failing to add hosts despite a working SSH connection
by Michel Jouvin 25 Oct '23

25 Oct '23

Hi, I'm struggling with a problem to add cephadm some hosts in our Quincy cluster. "ceph orch host add host addr" fails with the famous "missing 2 required positional arguments: 'hostname' and 'addr'" because of bug https://tracker.ceph.com/issues/59081 but looking at cephadm messages with "ceph -W cephadm", I can see: -------- Log: Opening SSH connection to 10.81.22.183, port 22 [conn=736] Connected to SSH server at 10.81.22.183, port 22 [conn=736] Local address: 10.81.22.151, port 53640 [conn=736] Peer address: 10.81.22.183, port 22 [conn=736] Login timeout expired [conn=736] Aborting connection Traceback (most recent call last): (removed) cephadm.ssh.HostConnectionError: Failed to connect to jc-rgw3 (10.81.22.183). Login timeout expired Log: Opening SSH connection to 10.81.22.183, port 22 [conn=736] Connected to SSH server at 10.81.22.183, port 22 [conn=736] Local address: 10.81.22.151, port 53640 [conn=736] Peer address: 10.81.22.183, port 22 [conn=736] Login timeout expired [conn=736] Aborting connection -------- It is very strange for me because " ssh -i /tmp/cephadm_identity_xxx 10.81.22.183" is working fine |when executed in the active mgr container. | |The host I'm trying to add is a RGW that has 3 active network connections: Ceph public network, our intranet network (used for managing the server) and the network of the application that will use the RGW. It seems to be somewhat related to this network configuration as main cluster servers (MONs, OSDs) which have only the the 2 Ceph networks and the intranet one don't suffer the same problem. In particular, what is strange is that I can successfully add the host if I use its intranet adress rather than the Ceph public network one (|||10.81.22.183) in the cephadm command. I have 3 hosts sharing the same network configuration and having the same problem. Any hint or suggestion to troubleshoot further this problem would be highly appreciated! Best regards, Michel

1 1

Re: radosgw - octopus - 500 Bad file descriptor on upload
by BEAUDICHON Hubert (Acoss) 25 Oct '23

25 Oct '23

Hi, We encountered the same kind of error for one of our users. CEPH Version : 16.2.10 2023-10-24T17:57:22.438+0200 7fc27ab44700 0 WARNING: set_req_state_err err_no=125 resorting to 500 2023-10-24T17:57:22.439+0200 7fc584957700 0 req 12200560481916573577 143.735748291s ERROR: RESTFUL_IO(s)->complete_header() returned err=Bad file descriptor 2023-10-24T17:57:22.439+0200 7fbecfaed700 1 ====== req done req=0x7fbdb86ab600 op status=-125 http_status=500 latency=143.735748291s ====== 2023-10-24T17:57:22.439+0200 7fbecfaed700 1 beast: 0x7fbdb86ab600: 10.227.131.117 - dev-centralog-save [24/Oct/2023:17:54:58.703 +0200] "PUT <obfuscate>&partNumber=1 HTTP/1.1" 500 58720313 - "<agent>" - latency=143.735748291s I haven't got any clue on the cause...

2 1

Ceph Leadership Team notes 10/25
by Dan van der Ster 25 Oct '23

25 Oct '23

Hi all, Here are this week's notes from the CLT: * Collective review of the Reef/Squid "State of Cephalopod" slides. * Smoke test suite was unscheduled but it's back on now. * Releases: * 17.2.7: about to start building last week, delayed by a few issues (https://tracker.ceph.com/issues/63257, https://tracker.ceph.com/issues/63305, https://github.com/ceph/ceph/pull/54169) ceph_exporter test coverage will be prioritized. * 18.2.1: all PRs in testing or merged. * Ceph Board approved a new Foundation member tiers model, Silver, Gold, Platinum, Diamond. Working on implementation with LF. -- dan

1 0

Ceph 16.2.14: pgmap updated every few seconds for no apparent reason
by Zakhar Kirpichenko 25 Oct '23

25 Oct '23

Hi, I am investigating excessive mon writes in our cluster and wondering whether excessive pgmap updates could be the culprit. Basically pgmap is updated every few seconds, sometimes over ten times per minute, in a healthy cluster with no OSD and/or PG changes: Oct 13 11:03:03 ceph03 bash[4019]: cluster 2023-10-13T11:03:01.515438+0000 mgr.ceph01.vankui (mgr.336635131) 838252 : cluster [DBG] pgmap v606575: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 60 MiB/s rd, 109 MiB/s wr, 5.65k op/s Oct 13 11:03:04 ceph03 bash[4019]: cluster 2023-10-13T11:03:03.520953+0000 mgr.ceph01.vankui (mgr.336635131) 838253 : cluster [DBG] pgmap v606576: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 64 MiB/s rd, 128 MiB/s wr, 5.76k op/s Oct 13 11:03:06 ceph03 bash[4019]: cluster 2023-10-13T11:03:05.524474+0000 mgr.ceph01.vankui (mgr.336635131) 838255 : cluster [DBG] pgmap v606577: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 64 MiB/s rd, 122 MiB/s wr, 5.57k op/s Oct 13 11:03:08 ceph03 bash[4019]: cluster 2023-10-13T11:03:07.530484+0000 mgr.ceph01.vankui (mgr.336635131) 838256 : cluster [DBG] pgmap v606578: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 79 MiB/s rd, 127 MiB/s wr, 6.62k op/s Oct 13 11:03:10 ceph03 bash[4019]: cluster 2023-10-13T11:03:09.533337+0000 mgr.ceph01.vankui (mgr.336635131) 838258 : cluster [DBG] pgmap v606579: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 66 MiB/s rd, 104 MiB/s wr, 5.38k op/s Oct 13 11:03:12 ceph03 bash[4019]: cluster 2023-10-13T11:03:11.537908+0000 mgr.ceph01.vankui (mgr.336635131) 838259 : cluster [DBG] pgmap v606580: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 85 MiB/s rd, 121 MiB/s wr, 6.43k op/s Oct 13 11:03:13 ceph03 bash[4019]: cluster 2023-10-13T11:03:13.543490+0000 mgr.ceph01.vankui (mgr.336635131) 838260 : cluster [DBG] pgmap v606581: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 78 MiB/s rd, 127 MiB/s wr, 6.54k op/s Oct 13 11:03:16 ceph03 bash[4019]: cluster 2023-10-13T11:03:15.547122+0000 mgr.ceph01.vankui (mgr.336635131) 838262 : cluster [DBG] pgmap v606582: 2400 pgs: 5 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 71 MiB/s rd, 122 MiB/s wr, 6.08k op/s Oct 13 11:03:18 ceph03 bash[4019]: cluster 2023-10-13T11:03:17.553180+0000 mgr.ceph01.vankui (mgr.336635131) 838263 : cluster [DBG] pgmap v606583: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 75 MiB/s rd, 176 MiB/s wr, 6.83k op/s Oct 13 11:03:20 ceph03 bash[4019]: cluster 2023-10-13T11:03:19.555960+0000 mgr.ceph01.vankui (mgr.336635131) 838264 : cluster [DBG] pgmap v606584: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 58 MiB/s rd, 161 MiB/s wr, 5.55k op/s Oct 13 11:03:22 ceph03 bash[4019]: cluster 2023-10-13T11:03:21.560597+0000 mgr.ceph01.vankui (mgr.336635131) 838266 : cluster [DBG] pgmap v606585: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 62 MiB/s rd, 221 MiB/s wr, 6.19k op/s Oct 13 11:03:24 ceph03 bash[4019]: cluster 2023-10-13T11:03:23.565974+0000 mgr.ceph01.vankui (mgr.336635131) 838267 : cluster [DBG] pgmap v606586: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 50 MiB/s rd, 246 MiB/s wr, 5.93k op/s Oct 13 11:03:26 ceph03 bash[4019]: cluster 2023-10-13T11:03:25.569471+0000 mgr.ceph01.vankui (mgr.336635131) 838269 : cluster [DBG] pgmap v606587: 2400 pgs: 1 active+clean+scrubbing, 5 active+clean+scrubbing+deep, 2394 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 41 MiB/s rd, 240 MiB/s wr, 4.99k op/s Oct 13 11:03:28 ceph03 bash[4019]: cluster 2023-10-13T11:03:27.575618+0000 mgr.ceph01.vankui (mgr.336635131) 838270 : cluster [DBG] pgmap v606588: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 44 MiB/s rd, 259 MiB/s wr, 5.38k op/s Oct 13 11:03:30 ceph03 bash[4019]: cluster 2023-10-13T11:03:29.578262+0000 mgr.ceph01.vankui (mgr.336635131) 838271 : cluster [DBG] pgmap v606589: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 31 MiB/s rd, 195 MiB/s wr, 4.06k op/s Oct 13 11:03:32 ceph03 bash[4019]: cluster 2023-10-13T11:03:31.582849+0000 mgr.ceph01.vankui (mgr.336635131) 838272 : cluster [DBG] pgmap v606590: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 35 MiB/s rd, 197 MiB/s wr, 4.43k op/s Oct 13 11:03:34 ceph03 bash[4019]: cluster 2023-10-13T11:03:33.588249+0000 mgr.ceph01.vankui (mgr.336635131) 838274 : cluster [DBG] pgmap v606591: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 39 MiB/s rd, 138 MiB/s wr, 4.50k op/s Oct 13 11:03:36 ceph03 bash[4019]: cluster 2023-10-13T11:03:35.591837+0000 mgr.ceph01.vankui (mgr.336635131) 838276 : cluster [DBG] pgmap v606592: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 35 MiB/s rd, 92 MiB/s wr, 3.90k op/s Oct 13 11:03:37 ceph03 bash[4019]: cluster 2023-10-13T11:03:37.597899+0000 mgr.ceph01.vankui (mgr.336635131) 838277 : cluster [DBG] pgmap v606593: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 42 MiB/s rd, 77 MiB/s wr, 4.47k op/s Oct 13 11:03:39 ceph03 bash[4019]: cluster 2023-10-13T11:03:39.600591+0000 mgr.ceph01.vankui (mgr.336635131) 838278 : cluster [DBG] pgmap v606594: 2400 pgs: 4 active+clean+scrubbing+deep, 2396 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 34 MiB/s rd, 39 MiB/s wr, 3.61k op/s Oct 13 11:03:41 ceph03 bash[4019]: cluster 2023-10-13T11:03:41.605546+0000 mgr.ceph01.vankui (mgr.336635131) 838279 : cluster [DBG] pgmap v606595: 2400 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 41 MiB/s rd, 42 MiB/s wr, 4.21k op/s Oct 13 11:03:43 ceph03 bash[4019]: cluster 2023-10-13T11:03:43.610694+0000 mgr.ceph01.vankui (mgr.336635131) 838281 : cluster [DBG] pgmap v606596: 2400 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 49 MiB/s rd, 49 MiB/s wr, 4.59k op/s Oct 13 11:03:45 ceph03 bash[4019]: cluster 2023-10-13T11:03:45.614721+0000 mgr.ceph01.vankui (mgr.336635131) 838283 : cluster [DBG] pgmap v606597: 2400 pgs: 1 active+clean+scrubbing, 4 active+clean+scrubbing+deep, 2395 active+clean; 16 TiB data, 61 TiB used, 716 TiB / 777 TiB avail; 44 MiB/s rd, 49 MiB/s wr, 4.15k op/s Within 14 hours of today pgmap was updated 19960 times for no apparent reason. The cluster is healthy and nothing is going on apart from normal I/O. Are frequent pgmap updates expected behavior? What may be causing these updates? I would very much appreciate your comments and suggestions. Best regards, Zakhar

3 5

Combining masks in ceph config
by Frank Schilder 25 Oct '23

25 Oct '23

Hi all, I have a case where I want to set options for a set of HDDs under a common sub-tree with root A. I have also HDDs in another disjoint sub-tree with root B. Therefore, I would like to do something like ceph config set osd/class:hdd,datacenter:A option value The above does not give a syntax error, but I'm also not sure it does the right thing. Does the above mean "class:hdd and datacenter:A" or does it mean "for OSDs with device class 'hdd,datacenter:A'"? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0