- ceph-users - lists.ceph.io

Re: RGW access logs with bucket name
by Boris Behrens 02 Nov '23

02 Nov '23

Bringing up that topic again: is it possible to log the bucket name in the rgw client logs? currently I am only to know the bucket name when someone access the bucket via https://TLD/bucket/object instead of https://bucket.TLD/object. Am Di., 3. Jan. 2023 um 10:25 Uhr schrieb Boris Behrens <bb(a)kervyn.de>: > Hi, > I am looking forward to move our logs from > /var/log/ceph/ceph-client...log to our logaggregator. > > Is there a way to have the bucket name in the log file? > > Or can I write the rgw_enable_ops_log into a file? Maybe I could work with > this. > > Cheers and happy new year > Boris > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

4 6

Emergency, I lost 4 monitors but all osd disk are safe
by Mohamed LAMDAOUAR 02 Nov '23

02 Nov '23

Hello, I have 7 machines on CEPH cluster, the service ceph runs on a docker container. Each machine has 4 hdd of data (available) and 2 nvme sssd (bricked) During a reboot, the ssd bricked on 4 machines, the data are available on the HDD disk but the nvme is bricked and the system is not available. is it possible to recover the data of the cluster (the data disk are all available)

7 14

"cephadm version" in reef returns "AttributeError: 'CephadmContext' object has no attribute 'fsid'"
by Martin Conway 02 Nov '23

02 Nov '23

Hi I wrote before about issues I was having with cephadm in 18.2.0 Sorry, I didn't see the helpful replies because my mail service binned the responses. I still can't get the reef version of cephadm to work properly. I had updated the system rpm to reef (ceph repo) and also upgraded the containerised ceph daemons to reef before my first email. Both the system package cephadm and the one found at /var/lib/ceph/${fsid}/cephadm.* return the same error when running "cephadm version" Traceback (most recent call last): File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9468, in <module> main() File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 9456, in main r = ctx.func(ctx) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2108, in _infer_image ctx.image = infer_local_ceph_image(ctx, ctx.container_engine.path) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2191, in infer_local_ceph_image container_info = get_container_info(ctx, daemon, daemon_name is not None) File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in get_container_info matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 2154, in <listcomp> matching_daemons = [d for d in daemons if daemon_name_or_type(d) == daemon_filter and d['fsid'] == ctx.fsid] File "./cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e", line 217, in __getattr__ return super().__getattribute__(name) AttributeError: 'CephadmContext' object has no attribute 'fsid' I am running into other issues as well, but I think they may point back to this issue of "'CephadmContext' object has no attribute 'fsid'" Any help would be appreciated. Regards, Martin Conway IT and Digital Media Manager Research School of Physics Australian National University Canberra ACT 2601 +61 2 6125 1599 https://physics.anu.edu.au<https://physics.anu.edu.au/>

4 8

diskprediction_local module and trained models
by Can Özyurt 02 Nov '23

02 Nov '23

Hi everyone, We have recently noticed diskprediction_local module only works for a set of manufacturers. Hence we have the following questions: Are there any plans to support more manufacturers in the near future? Can we contribute to the process of training new models and how? Can the existing models be used with disks of other vendors (with hackish methods)? Is it that It just doesn't work or it would not be reliable? Basically how does the trained model and the vendor relation work? Say we have a disk for an existing model, what is the best way to test the trained model and the module itself? Thanks in advance

1 0

CephFS scrub causing MDS OOM-kill
by Denis Polom 02 Nov '23

02 Nov '23

Hi, I did setup CephFS forward scrub by executing cmd # ceph tell mds.cephfs:0 scrub start / recursive { "return_code": 0, "scrub_tag": "37a67f72-89a3-474e-8f8b-1e55cb979e14", "mode": "asynchronous" } But immediately after it started, memory usage on MDS that keeps rank 0 increased from 2GB to 32GB until MDS was killed by OOM killer. Why does it consume all the memory? Can it be adjusted somehow? Thank you

1 0

Ceph Leadership Team Meeting: 2023-11-1 Minutes
by Casey Bodley 01 Nov '23

01 Nov '23

quincy 17.2.7: released! * major 'dashboard v3' changes causing issues? https://github.com/ceph/ceph/pull/54250 did not merge for 17.2.7 * planning a retrospective to discuss what kind of changes should go in minor releases when members of the dashboard team are present reef 18.2.1: * most PRs already tested/merged * possibly start validation next week?

1 0

Moving devices to a different device class?
by Matt Larson 01 Nov '23

01 Nov '23

I am looking to create a new pool that would be backed by a particular set of drives that are larger nVME SSDs (Intel SSDPF2NV153TZ, 15TB drives). Particularly, I am wondering about what is the best way to move devices from one pool and to direct them to be used in a new pool to be created. In this case, the documentation suggests I could want to assign them to a new device-class and have a placement rule that targets that device-class in the new pool. Currently the Ceph cluster has two device classes 'hdd' and 'ssd', and the larger 15TB drives were automatically assigned to the 'ssd' device class that is in use by a different pool. The `ssd` device classes are used in a placement rule targeting that class. The documentation describes that I could set a device class for an OSD with a command like: `ceph osd crush set-device-class CLASS OSD_ID [OSD_ID ..]` Class names can be arbitrary strings like 'big_nvme". Before setting a new device class to an OSD that already has an assigned device class, should use `ceph osd crush rm-device-class ssd osd.XX`. Can I proceed to directly remove these OSDs from the current device class and assign to a new device class? Should they be moved one by one? What is the way to safely protect data from the existing pool that they are mapped to? Thanks, Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

4 5

negative list operation causing degradation in performance
by Vitaly Goot 01 Nov '23

01 Nov '23

In Ceph version 17.2.5, a notable performance drop is observed when executing list operations on non-existent objects, especially for large buckets undergoing high concurrency with frequent updates. For the context, initiating concurrent PUT/DELETE/LIST operations on 4 buckets, each containing ~10 million objects, initially yields satisfactory latencies. However, as updates continue to accumulate, the efficiency of the `negative list` operation diminishes notably. I suspect that over time, with many object insertions and deletions, the bucket index gets fragmented. While this affects all operations, negative lookups appear to be particularly hampered, possibly because they necessitate a comprehensive scan of the fragmented index. With the LIST arguments defined as $prefix=xxx $max-keys=2, are there any recommendations to optimize the Ceph configuration to enhance the performance of negative list operations?

1 0

find PG with large omap object
by Frank Schilder 31 Oct '23

31 Oct '23

Hi all, we had a bunch of large omap object warnings after a user deleted a lot of files on a ceph fs with snapshots. After the snapshots were rotated out, all but one of these warnings disappeared over time. However, one warning is stuck and I wonder if its something else. Is there a reasonable way (say, one-liner with no more than 120 characters) to get ceph to tell me which PG this is coming from? I just want to issue a deep scrub to check if it disappears and going through the logs and querying every single object for its key count seems a bit of a hassle for something that ought to be part of "ceph health detail". Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

Add nats_adapter
by Vahideh Alinouri 30 Oct '23

30 Oct '23

Dear Ceph Users, I am requesting the backporting changes related to the nats_adapter.lua. This feature is in a version newer than pacific, but we don't have it in pacific version. I would greatly appreciate it if someone from the Ceph development team backport this change to the pacific version. Best regards, Vahideh Alinouri

2 1