- ceph-users - lists.ceph.io

Reef osd_memory_target and swapping
by Dave Hall 23 Oct '24

23 Oct '24

Hello. I'm seeing the following in the Dashboard -> Configuration panel for osd_memory_target: Default: 4294967296 Current Values: osd: 9797659437, osd: 10408081664, osd: 11381160192, osd: 22260320563 I have 4 hoists in the cluster right now - all OSD+MGR+MON. 3 have 128GB RAM, the 4th has 256GB. On the host with 256GB, top shows some OSD processes with very high VIRT and RES values - the highest VIRT OSD has 13.0g. The highest RES is 8.5g. All 4 systems are currently swapping, but the 256GB system has much higher swap usage. I am confused why I have 4 current values for osd_memory_target, and especially about the 4th one at 22GB. Also, I'm recalling that there might be a recommendation to disable swap. and I could easily do 'swapoff -a' when the swap usage is lower than the free RAM. Can anybody shed any light on this? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

7 8

Squid Manager Daemon: balancer crashing orchestrator and dashboard
by laimis.juzeliunas＠oxylabs.io 22 Oct '24

22 Oct '24

Hello community, We are facing one issue after migrating from Reef 18.4.2 to Squid 19.2.0 with the Ceph manager daemon and were wondering if anyone has already faced this or could guide us where to look further. When turning on balancer (upmap mode) it hangs our mgr completely most of the time and leaves orchestrator as well as the dashboard/UI unresponsive. We noticed this initially during the upgrade (as mgr is the first in line) and had to continue with the balancer turned off. Post upgrade we still need to keep it turned off. We run docker daemons with cephadm as the orchestrator, health status is always HEALTH_OK. A few observations: ceph balancer on -> mgr logs show some pg_upmap's being performed debug logs show that balancer is done -> mgr stops working only pgmap debug logs remain on container after some (10-20 minutes) time mgr fails over to standby node mgr starts and begins full cluster inventory (services, disks, daemons, networks, etc) dashboard starts up, orch commands begin working balancer kicks in, performs some upmaps and the cycle continues failing over to standby node afrer quite some time while mgr is not working TCP port is showing as being listened to (netstat), but does not even respond to telnet. but when mgr is working - we can query it with curl Cluster overview: cluster: id: 96df99f6-fc1a-11ea-90a4-6cb3113cb732 health: HEALTH_OK services: mon: 5 daemons, quorum ceph-node004,ceph-node003,ceph-node001,ceph-node005,ceph-node002 (age 3d) mgr: ceph-node001.hgythj(active, since 24h), standbys: ceph-node002.jphtvg mds: 21/21 daemons up, 12 standby osd: 384 osds: 384 up (since 2d), 384 in (since 7d) rbd-mirror: 2 daemons active (2 hosts) rgw: 64 daemons active (32 hosts, 1 zones) data: volumes: 2/2 healthy pools: 16 pools, 12793 pgs objects: 751.20M objects, 1.4 PiB usage: 4.5 PiB used, 1.1 PiB / 5.6 PiB avail pgs: 12410 active+clean 285 active+clean+scrubbing 98 active+clean+scrubbing+deep io: client: 5.8 GiB/s rd, 169 MiB/s wr, 43.77k op/s rd, 9.92k op/s wr We will be able to provide a more detailed log sequence but for now we see these entries: debug 2024-10-18T13:23:34.478+0000 7fad971ab640 0 [balancer DEBUG root] Waking up [active, now 2024-10-18_13:23:34] debug 2024-10-18T13:23:34.478+0000 7fad971ab640 0 [balancer DEBUG root] Running debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] Optimize plan auto_2024-10-18_13:23:34 debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] Mode upmap, max misplaced 0.050000 debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer DEBUG root] unknown 0.000000 degraded 0.000000 inactive 0.000000 misplaced 0 debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] do_upmap debug 2024-10-18T13:23:34.730+0000 7fad971ab640 0 [balancer INFO root] pools ['cephfs.our-awesome-pool1.meta', 'our-awesome-k8s.ceph-csi.rbd', 'cephfs.our-awesome-poo2l.data', 'cephfs.our-awesome-pool2.meta', 'europe-1.rgw.buckets.non-ec', 'europe-1.rgw.buckets.data', 'our-awesome-k8s2.ceph-csi.rbd', 'europe-1.rgw.log', '.mgr', 'our-awesome-k8s4.ceph-csi.rbd', 'europe-1.rgw.control', 'our-awesome-k8s3.ceph-csi.rbd', 'europe-1.rgw.meta', 'europe-1.rgw.buckets.index', '.rgw.root', 'cephfs.our-awesome-pool1.data'] debug 2024-10-18T13:23:34.814+0000 7fad971ab640 0 [balancer INFO root] prepared 10/10 upmap changes debug 2024-10-18T13:23:34.814+0000 7fad971ab640 0 [balancer INFO root] Executing plan auto_2024-10-18_13:23:34 debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.148 mappings [{'from': 113, 'to': 94}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.302 mappings [{'from': 138, 'to': 128}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.34e mappings [{'from': 92, 'to': 89}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.504 mappings [{'from': 156, 'to': 94}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.836 mappings [{'from': 148, 'to': 54}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.e4c mappings [{'from': 157, 'to': 54}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.e56 mappings [{'from': 147, 'to': 186}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.e63 mappings [{'from': 79, 'to': 31}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.ed9 mappings [{'from': 158, 'to': 237}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer INFO root] ceph osd pg-upmap-items 44.f1e mappings [{'from': 153, 'to': 237}] debug 2024-10-18T13:23:34.818+0000 7fad971ab640 0 [balancer DEBUG root] commands [<mgr_module.CommandResult object at 0x7fae2e05b700>, <mgr_module.CommandResult object at 0x7fad2d889ca0>, <mgr_module.CommandResult object at 0x7fad2d810340>, <mgr_module.CommandResult object at 0x7fad2d88c5e0>, <mgr_module.CommandResult object at 0x7fad2d418130>, <mgr_module.CommandResult object at 0x7fad2d418ac0>, <mgr_module.CommandResult object at 0x7fae2f8d7cd0>, <mgr_module.CommandResult object at 0x7fad2d448520>, <mgr_module.CommandResult object at 0x7fad2d4480d0>, <mgr_module.CommandResult object at 0x7fad2d448fa0>] 162.55.93.25 - - [18/Oct/2024:13:23:35] "GET /metrics HTTP/1.1" 200 7731011 "" "Prometheus/2.43.0" ... debug 2024-10-18T13:23:36.110+0000 7fad971ab640 0 [balancer DEBUG root] done We were suspecting that this might be caused by older ceph-clients connecting and getting identified (wrongly) as Luminous by Ceph due to the fresh upmap-read balancer mode (I think it came with Squid but I might be wrong) that had to do something in the background even when disabled. However setting set-require-min-compat-client did not help our case so we dismissed this assumption. We would be beyond happy for any advice where to look further as having no balancer is sad. If anyone would like to go through detailed logs we are glad to provide. Best, Laimis J.

3 2

CSC Election: Governance amendments and Ceph Executive Council Nominations
by Patrick Donnelly 22 Oct '24

22 Oct '24

The Ceph Steering Committee is voting on a number of governance amendments and nominations to the Ceph Executive Council. The election is public and viewable here: https://vote.heliosvoting.org/helios/elections/e03494ce-e04c-41d0-bb05-ec5c… The election closes on October 28th, 3pm UTC. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Ceph RGW performance guidelines
by Harry Kominos 21 Oct '24

21 Oct '24

Hello Ceph Community! I have the following very interesting problem, for which I found no clear guidelines upstream so I am hoping to get some input from the mailing list. I have a 6PB cluster in operation which is currently half full. The cluster has around 1K OSD, and the RGW data pool has 4096 pgs (and pgp_num). The issue is as follows: Let's say that we have 10 million small objects (4MB) each. 1)Is there a performance difference *when fetching* between storing all 10 million objects in one bucket and storing 1 million in 10 buckets? There should be "some" because of the different number of pgs in use, in the 2 scenarios but it is very hard to quantify. 2) What if I have 100 million objects? Is there some theoretical limit / guideline on the number of objects that I should have in a bucket before I see performance drops? I should mention here that the contents of the bucket *never need to be listed, *The user always knows how to do a curl, to get the contents. Thank you for your help, Harry P.S. The following URLs have been very informative, but they do not answer my question unfortunately. https://www.redhat.com/en/blog/red-hat-ceph-object-store-dell-emc-servers-p… https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond

2 5

How to Speed Up Draining OSDs?
by Alex Hussein-Kershaw (HE/HIM) 21 Oct '24

21 Oct '24

Hi Folks, I'm trying to scale-in a Ceph Cluster. It's running 19.2.0 and is cephadm managed. It's just a test system, so has basically no data and only has 3 OSDs. As part of the scaling-in, I run "ceph orch host drain <hostname> --zap-osd-devices" as per Host Management — Ceph Documentation<https://docs.ceph.com/en/reef/cephadm/host-management/#removing-hosts>. That starts off the OSD draining. However, that drain seems to take an enormous amount of time. My OSD has less than 100MiB raw storage, and I let it run for 2 hours over lunch and it still was not finished, so I cancelled it. I'm not sure how this scales, but I'm assuming at least linearly with data stored, which seems like bad news for doing this on real systems, which may have several TBs per OSD. I had a look at the recovery profiles documentation here mClock Config Reference — Ceph Documentation<https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/> which seemed to indicate I could speed this up (but my impression was maybe I could get a speed up of 2x which seems like it will still take an age). On the other hand, just switching off the host running the OSD and doing an offline host removal ("ceph orch host rm <hostname> --offline") seems much easier, with the trade-off that the Cluster recovers after the loss of the OSD rather than pre-emptively. But that big risk of that seems to be mitigated by "ceph orch host ok-to-stop <hostname>" to check I won't cause any PGs to go offline before hand. Are there any tricks here that I'm missing? Thanks, Alex

2 4

Unable to mount NFS share NFSv3 on windows client.
by Saif Mohammad 21 Oct '24

21 Oct '24

Hello Everyone, We are encountering issues while trying to mount NFSv3-based exports on a Windows 11 client. We made the necessary changes to the configuration file to enable NFSv3 exports, but unable to mount. Below are the details: We created the custom config file named export.conf and applied the changes as follow. ------ { "export_id": 1, "path": "/volumes", "cluster_id": "nfs-cephfs", "pseudo": "/ceph", "access_type": "RW", "squash": "root_squash", "security_label": false, "Enable_UDP": false, "protocols": [ 3 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "cephfs01" }, "clients": [] } ------ On Client machine (Windows 11), we attempted to mount the NFS share using the following command: ------ \Users\Administrator>mount 172.16.100.14:/mnt/share S: Network Error - 53 Type 'NET HELPMSG 53' for more information. ------ we verified there is no issue with the network connectivity, For reference we are sharing the rpcinfo output: ------ C:\Users\Administrator>rpcinfo -p 192.168.2.107 program version protocol port (null) (null) -------------------------------------------------- 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100003 4 udp 2049 nfs 100003 4 tcp 2049 nfs ------ kindly assist us in identifying what might be missing or suggest any troubleshooting steps? Your assistance will be greatly appreciated. Kind regards, Mohammad Saif

1 0

Re: Ubuntu 24.02 LTS Ceph status warning
by Eugen Block 18 Oct '24

18 Oct '24

Is apparmor configured differently on those hosts? Or is it running only on the misbehaving host? Zitat von Dominique Ramaekers <dominique.ramaekers(a)cometal.be>: > 'ceph config get mgr container_image' gives > quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a => > OK > > 'ceph health detail' gives > HEALTH_WARN failed to probe daemons or devices > [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices > host hvs004 `cephadm gather-facts` failed: cephadm exited with > an error code: 1, stderr: Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in > <module> > ... > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in > _fetch_apparmor > ValueError: too many values to unpack (expected 2) > host hvs004 `cephadm ceph-volume` failed: cephadm exited with an > error code: 1, stderr: Inferring config > /var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/config/ceph.conf > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/__main__.py", line 5579, in > <module> > ... > File > "/var/lib/ceph/dd4b0610-b4d2-11ec-bb58-d1b32ae31585/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/host_facts.py", line 722, in > _fetch_apparmor > ValueError: too many values to unpack (expected 2) > > I do think it's a ceph version issue, so I started to compaire the > hvs004 host with a good behaving host hvs001. I did find this: > 'root@hvs001~#:cephadm shell ceph -v' gives > ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable) > 'root@hvs004~#:cephadm shell ceph -v' gives > ceph version 19.3.0-5346-gcc481a63 > (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev) > > It seems only the shell uses a wrong docker image so I took a list > from both of them: > hvs0001 > ---------- > REPOSITORY TAG IMAGE ID CREATED > SIZE > quay.io/ceph/ceph v19 37996728e013 2 weeks > ago 1.28GB > quay.io/ceph/ceph v18.2 2bc0b0f4375d 2 months > ago 1.22GB > quay.io/ceph/ceph v18 a27483cc3ea0 6 months > ago 1.26GB > quay.io/ceph/ceph v17 5a04c8b3735d 9 months > ago 1.27GB > quay.io/ceph/ceph-grafana 9.4.7 954c08fa6188 10 > months ago 633MB > quay.io/ceph/grafana 9.4.12 2bacad6d85d8 17 > months ago 330MB > quay.io/prometheus/prometheus v2.43.0 a07b618ecd1d 19 > months ago 234MB > quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 22 > months ago 65.1MB > quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 22 > months ago 22.5MB > quay.io/ceph/ceph v17.2 0912465dcea5 2 years > ago 1.34GB > quay.io/ceph/ceph v17.2.3 0912465dcea5 2 years > ago 1.34GB > quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years > ago 558MB > quay.ceph.io/ceph-ci/ceph master c5ce177c6a5d 2 years > ago 1.38GB > quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 2 years > ago 204MB > quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years > ago 20.9MB > quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 3 years > ago 57.5MB > quay.io/ceph/ceph-grafana 6.7.4 557c83e11646 3 years > ago 486MB > quay.io/prometheus/prometheus v2.18.1 de242295e225 4 years > ago 140MB > quay.io/prometheus/alertmanager v0.20.0 0881eb8f169f 4 years > ago 52.1MB > quay.io/prometheus/node-exporter v0.18.1 e5a616e4b9cf 5 years > ago 22.9MB > > hvs004 > --------- > REPOSITORY TAG IMAGE ID CREATED > SIZE > quay.ceph.io/ceph-ci/ceph main 6e76ca06f33a 11 days > ago 1.41GB > quay.io/ceph/ceph <none> 37996728e013 2 weeks > ago 1.28GB > quay.io/ceph/ceph v17 9cea3956c04b 18 > months ago 1.16GB > quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 22 > months ago 22.5MB > quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years > ago 20.9MB > > I pulled on hvs004 the v19 tagged image and my cephadm shell ceph -v > gave the correct version. > > It seems my docker images aren't automatically managed by ceph? > > Can I fix this, or do I have to pull the correct images and remove > the wrong ones myself? > > >> -----Oorspronkelijk bericht----- >> Van: Eugen Block <eblock(a)nde.ag> >> Verzonden: vrijdag 11 oktober 2024 13:03 >> Aan: ceph-users(a)ceph.io >> Onderwerp: [ceph-users] Re: Ubuntu 24.02 LTS Ceph status warning >> >> I don't think the warning is related to a specific ceph version. The >> orchestrator uses the default image anyway, you can get it via: >> >> ceph config get mgr container_image >> >> 'ceph health detail' should reveal which host or daemons misbehaves. I >> would then look into cephadm.log on that host to find more hints, what >> exactly goes wrong. You should also look into the active MGR log, >> it could also >> give you hints why that service fails. >> >> Zitat von Dominique Ramaekers <dominique.ramaekers(a)cometal.be>: >> >> > I manage a 4 hosts cluster on Ubuntu 22.04 LTS with ceph installed >> > trough cephad and containers on Docker. >> > >> > Last month, I've migrated to the latest Ceph 19.2. All went great. >> > >> > Last week I've upgraded one of my hosts to Ubuntu 24.04.1 LTS Now I >> > get the following warning in cephadm shell -- ceph status: >> > Failed to apply 1 service(s): osd.all-available-devices failed to >> > probe daemons or devices >> > >> > Outside the ceph shell: >> > Ceph -v results in 'ceph version 19.2.0~git20240301.4c76c50 >> > (4c76c50a73f63ba48ccdf0adccce03b00d1d80c7) squid (dev)' >> > >> > Inside the shell: 'ceph version 19.3.0-5346-gcc481a63 >> > (cc481a63bc03a534cb8e2e961293d6509ba59401) squid (dev)' >> > All osd's, mon's , mgr's and mds's are on 19.2.0` (image id >> > 37996728e013) >> > >> > Do I get the warning because the Ubuntu package of ceph is still on a >> > development version? >> > Or can I have another underling problem? >> > >> > Thanks for the help. >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> > email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email >> to ceph-users-leave(a)ceph.io

2 2

Procedure for temporary evacuation and replacement
by Frank Schilder 18 Oct '24

18 Oct '24

Hi all, a hopefully simple question this time. I would like a second opinion on a procedure for replacing a larger number of disks. We need to replace about 40 disks distributed over all 12 hosts backing a large pool with EC 8+3. We can't do it host by host as it would take way too long (replace disks per host and let recovery rebuild the data). Therefore, we would like to evacuate all data from these disks simultaneously and with as little data movement as possible. This is the procedure that seems to do the trick: 1.) For all OSDs: ceph osd reweight ID 0 # Note: not "osd crush reweight" 2.) Wait for rebalance to finish 3.) Replace disks and deploy OSDs with the same IDs as before per host 4.) Start OSDs and let rebalance back I tested step 1 on Octopus with 1 disk and it seems to work. The reason I ask is that step 1 actually marks the OSDs as OUT. However, they are still UP and I see only misplaced objects, not degraded objects. It is a bit counter-intuitive, but it seems that UP+OUT OSDs still participate in IO. Because it is counter-intuitive, I would like to have a second opinion. I have read before that others reweight to something like 0.001 and hope that this flushes all PGs. I would prefer not to rely on hope and a reweight to 0 apparently is a valid choice here, leading to a somewhat weird state with UP+OUT OSDs. Problems that could arise are timeouts I'm overlooking that will make data chunks on UP+OUT OSDs unavailable after some time. I'm also wondering if UP+OUT OSDs participate in peering in case there is an OSD restart somewhere in the pool. Thanks for your input and best regards! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

7 16

Lifecycle Stuck PROCESSING and UNINITIAL
by Reid Guyett 18 Oct '24

18 Oct '24

Hello, I am experiencing an issue where it seems all lifecycles are showing either PROCESSING or UNINITIAL. > # radosgw-admin lc list > [ > { > "bucket": > ":tesra:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.833499554.20", > "shard": "lc.0", > "started": "Thu, 17 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":primevideos:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.311333334.886", > "shard": "lc.3", > "started": "Wed, 16 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":editorimages:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.311333334.3", > "shard": "lc.4", > "started": "Wed, 16 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1", > "shard": "lc.10", > "started": "Thu, 17 Oct 2024 00:00:01 GMT", > "status": "PROCESSING" > }, > { > "bucket": > ":projects0609:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.856877269.1147", > "shard": "lc.10", > "started": "Thu, 01 Jan 1970 00:00:00 GMT", > "status": "UNINITIAL" > }, > ... > I turned up the log level for rgw and see a most have some sort of error: "failed to put head" or "returned error ret==-2" > 2024-10-17T15:08:31.720+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.10 head last stored at Mon Sep > 23 00:00:00 2024 > 2024-10-17T15:08:31.720+0000 7fc9cf968640 16 lifecycle: > RGWLC::expired_session started: 1729123201 interval: 86400(*2==172800) now: > 1729177711 > 2024-10-17T15:08:31.720+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ACTIVE entry: > :osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1::1729123201:1 > index: 10 worker ix: 0 > 2024-10-17T15:08:31.744+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > failed to put head lc.10 > 2024-10-17T15:08:31.779+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 2 worker ix: 0 > 2024-10-17T15:08:31.781+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.2 head last stored at Wed Jun > 19 00:00:02 2024 > 2024-10-17T15:08:31.781+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.782+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 23 worker ix: 0 > 2024-10-17T15:08:31.783+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.23 head last stored at Mon Jan > 8 00:00:00 2024 > 2024-10-17T15:08:31.784+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.784+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 22 worker ix: 0 > 2024-10-17T15:08:31.786+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.22 head last stored at Mon Aug > 19 00:00:00 2024 > 2024-10-17T15:08:31.786+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.787+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 17 worker ix: 0 > 2024-10-17T15:08:31.788+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.17 head last stored at Mon Jul > 22 00:00:00 2024 > 2024-10-17T15:08:31.788+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.789+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 7 worker ix: 0 > 2024-10-17T15:08:31.790+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.7 head last stored at Sat Jun > 8 00:00:02 2024 > 2024-10-17T15:08:31.791+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 > 2024-10-17T15:08:31.791+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ENTER: index: 0 worker ix: 0 > 2024-10-17T15:08:31.793+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > head.marker !empty() at START for shard==lc.0 head last stored at Mon Sep > 23 00:00:00 2024 > 2024-10-17T15:08:31.793+0000 7fc9cf968640 16 lifecycle: > RGWLC::expired_session started: 1729123201 interval: 86400(*2==172800) now: > 1729177711 > 2024-10-17T15:08:31.793+0000 7fc9cf968640 5 lifecycle: RGWLC::process(): > ACTIVE entry: > :tesra:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.833499554.20::1729123201:1 > index: 0 worker ix: 0 > 2024-10-17T15:08:31.794+0000 7fc9cf968640 0 lifecycle: RGWLC::process() > failed to put head lc.0 > 2024-10-17T15:08:31.795+0000 7fc9cf968640 2 lifecycle: life cycle: stop > I don't see any errors when running the process command but the status doesn't update to COMPLETE: > # radosgw-admin lc process --bucket osbackup --debug_rgw 10/5 > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 Realm: realname > (5e3b0d1c-78de-499a-9fa3-23e2f9142ef7) > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 ZoneGroup: us > (1c370840-4d35-4ada-85a4-14bd6eeb2b0a) > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 Zone: zonename > (5e9bc383-f7bd-4fd1-b607-1e563bfe0011) > 2024-10-17T15:13:32.826+0000 7f59f15c0840 4 using period configuration: > 69f026df-8fa4-42c8-8f62-1b0a902db797:12 > 2024-10-17T15:13:32.900+0000 7f59f15c0840 2 all 8 watchers are set, > enabling cache > 2024-10-17T15:13:32.923+0000 7f59d7536640 2 rgw data changes log: > RGWDataChangesLog::ChangesRenewThread: start > 2024-10-17T15:13:33.609+0000 7f59f15c0840 10 rgw notify: Started > notification manager with: 1 workers > 2024-10-17T15:13:33.610+0000 7f59f15c0840 10 cache get: > name=zonename.rgw.meta+root+osbackup : miss > 2024-10-17T15:13:33.611+0000 7f59f15c0840 10 cache put: > name=zonename.rgw.meta+root+osbackup info.flags=0x11 > 2024-10-17T15:13:33.611+0000 7f59f15c0840 10 > adding zonename.rgw.meta+root+osbackup to cache LRU end > 2024-10-17T15:13:33.611+0000 7f59f15c0840 10 cache get: > name=zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > : miss > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 cache put: > name=zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > info.flags=0x17 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 > adding zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > to cache LRU end > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 updating xattr: > name=ceph.objclass.version bl.length()=42 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 updating xattr: > name=user.rgw.acl bl.length()=153 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 updating xattr: name= > user.rgw.lc bl.length()=208 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 10 chain_cache_entry: > cache_locator=zonename.rgw.meta+root+.bucket.meta.osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1 > 2024-10-17T15:13:33.612+0000 7f59f15c0840 5 lifecycle: > RGWLC::process_bucket(): ENTER: index: 10 worker ix: 0 > 2024-10-17T15:13:33.641+0000 7f59f15c0840 5 lifecycle: > RGWLC::process_bucket(): ACTIVE entry: > :osbackup:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.668063260.1::1729123201:1 > index: 10 worker ix: 0 > 2024-10-17T15:13:33.815+0000 7f59d7536640 2 rgw data changes log: > RGWDataChangesLog::ChangesRenewThread: start > 2024-10-17T15:13:33.867+0000 7f59f15c0840 2 removed watcher, disabling > cache > I do see that one of the buckets doesn't exist but still is in the lc list. > # radosgw-admin lc list | grep projects0609 > "bucket": > ":projects0609:5e9bc383-f7bd-4fd1-b607-1e563bfe0011.856877269.1147", > # radosgw-admin bucket stats --bucket projects0609 > failure: (2002) Unknown error 2002: Is it possible to rm an LC entry from the LC list if the bucket doesn't exist? Is there anything else to check? I saw https://tracker.ceph.com/issues/68160 and the related tickets in the comments. Maybe it is related. Our RGWs are running in 18.2.4 official containers but our cluster is 17.2.7 debian packages. We upgraded the RGWs due to constant crashing. Thanks, Reid

2 1

Help with "27 osd(s) are not reachable" when also "27 osds: 27 up.. 27 in"
by Harry G Coin 17 Oct '24

17 Oct '24

I need help to remove a useless "HEALTH ERR" in 19.2.0 on a fully dual stack docker setup with ceph using ip v6, public and private nets separated, with a few servers. After upgrading from an error free v18 rev, I can't get rid of the 'health err' owing to the report that all osds are unreachable. Meanwhile ceph -s reports all osds up and in and the cluster otherwise operates normally. I don't care if it's 'a real fix' I just need to remove the false error report. Any ideas? Thanks Harry Coin

5 7