- ceph-users - lists.ceph.io

Orphaned rbd_data Objects
by Stolte, Felix 05 Feb '25

05 Feb '25

Hi guys, we have a rbd pool we used for images exported via ceph-iscsi on a 17.2.7 cluster. The pool uses 10 times the diskspace i would suppose it should and after investigating we noticed a lot of rbd_data Objects which images are no longer present. I assume that the original images were deleted using the gwcli but not all Objects have been removed properly. What would be the best/most secure way to get rid of these orphaned objects and reclaim the diskspace? Best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

4 11

ceph-osd segmentation fault on arm64 quay.io/ceph/ceph:v18.2.4
by Adam Prycki 05 Feb '25

05 Feb '25

Hi, today we attempted to upgrade our ceph cluster from 18.2.0 to 18.2.4. OSDs updated to 18.2.4 couldn't start. I've discovered that ceph-osd in segfaults. It happens always on our setup. I've tested it on ubunut 20.04 with docker and ubuntu 22.04 with podman Example: root@ktsh02:~# podman run --rm -it quay.io/ceph/ceph:v18.2.4 /bin/bash [root@20af860446bd /]# ceph-osd Segmentation fault (core dumped) 19.2.0 container seems to be fine. Do we have other users with arm64 ceph deployments? Do 18.2.4 containers work on your machines? Best regards Adam Prycki

2 1

Spec file: Possible typo in example:
by Alan Murrell 05 Feb '25

05 Feb '25

Hello, I am not sure if this is the right list to put this, but I was just looking over the documentation for the Service Spec file: https://docs.ceph.com/en/octopus/cephadm/drivegroups/ and under one of the "Advanced Cases" example, I believe there *might* be an error? Here is the example (for 20 HDDs, 12 SSDs, 2 NVMEs): --- START --- service_type: osd service_id: osd_spec_hdd placement: host_pattern: '*' data_devices: rotational: 0 db_devices: model: MC-55-44-XZ limit: 2 (db_slots is actually to be favoured here, but it's not implemented yet) --- service_type: osd service_id: osd_spec_ssd placement: host_pattern: '*' data_devices: model: MC-55-44-XZ db_devices: vendor: VendorC --- END --- For the 'data_devices' under the 'osd_spec_hdd' layout, should 'rotational' not be '1' instead of '0'? The earlier "Simple" example (20 HDDs, 2 SSDs) has 'rotational' as '1'.

2 4

Ceph User Stories Survey: Final Call
by Laura Flores 05 Feb '25

05 Feb '25

Hi all, Many thanks to everyone who has already taken the "Ceph User Stories" survey, the results of which will be discussed in one of the upcoming User + Dev Monthly Meetups. See https://www.meetup.com/ceph-user-group/events/ for updates on upcoming meetups. The survey will be closed this week on Friday, February 7th at 5pm CST. Until then, you may still take the survey by following this link: https://docs.google.com/forms/d/e/1FAIpQLSe66NedXh4gHLgk9G45eqP5V2wHlz4IKqR… Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: cephadm: Move DB/WAL from HDD to SSD
by Alan Murrell 04 Feb '25

04 Feb '25

One last question/clarification: when it becomes necessary to replace that SSD, I assume the process of moving the DB/WAL to the new SSD is the same, except in the 'ceph-volume migrate' command I would specify '--from db' now, yes? Sent from my mobile device. Please excuse brevity and ttpos. On Feb 4, 2025 00:18, Eugen Block <eblock(a)nde.ag> wrote: *** This is an EXTERNAL email. Please exercise caution. DO NOT open attachments or click links from unknown senders or unexpected email. *** I'm glad it worked. I thought those steps had already been added to the docs (cephadm specific), but I couldn't find them either. I'll ping Zac about it. Zitat von Alan Murrell <Alan(a)t-net.ca>: > OK, I think I am good now. I have completed the rest of the steps > (exit shell, start the osd.10 daemon back up again, waited for it to > be marked as "Up") and then I ran: > > ceph osd metadata 10 > > and all the bluestore_db items are pointing to the SSD now. > > OK, so now to note these steps/commands in to "Ceph Notes" and the > repeat the process for the other HDDs on this node and my other four 😊 > > Thank you *so* much for your help (and patience!) > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Feb 3 Ceph Steering Committee meeting notes
by Casey Bodley 04 Feb '25

04 Feb '25

discussed timeline for Tentacle release: - debated freeze soon (spring) or later (summer) to match Squid's delayed release - polled dev leads: later release preferred for EC optimizations and bluestore improvements - potential conflict with sepia lab migration which needs to be finished by end of year - tentative consensus for delaying Tentacle release to early 2026. further emails to follow on recent disruption from windows pr check: - broke over the weekend, fixed promptly with https://tracker.ceph.com/issues/69768 https://github.com/ceph/ceph/pull/61627 - future of windows support is uncertain, but we'll keep the windows check 'required' as long as it's maintained 19.2.1 release status: - packages/containers built, LRC upgrade planned CDM this week: - Please add topics: https://tracker.ceph.com/projects/ceph/wiki/CDM_05-FEB-2025 question about policy for AI-generated code contributions: - Linux Foundation policy at https://www.linuxfoundation.org/legal/generative-ai - the Ceph project should discuss

2 1

Full-Mesh?
by Anthony Fecarotta 04 Feb '25

04 Feb '25

I read a Ceph Benchmark paper ( https://www.proxmox.com/en/downloads/proxmox-virtual-environment/documentat…) where they demonstrated, among other things, the performance of using a Full-Mesh Network Schema for Ceph on a three node cluster. > • FRR ! full-mesh routed (with fallback) > • RSTP ! full-mesh RSTP loop setup > • Routed: ! full-mesh simple routed setup with round-robin bonds as > underlying links Is this method used at an enterprise level? I am seriously considering it. Thank you, everyone. Regards, * Anthony Fecarotta* Founder & President [image: phone-icon] anthony(a)linehaul.ai [image: phone-icon] 224-339-1182 [image: phone-icon] (855) 625-0300 [image: phone-icon] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image: phone-icon] www.linehaul.ai [image: linehaul.ai logo] <http://www.linehaul.ai/> [image: linkedin icon] <https://www.linkedin.com/in/anthony-fec/>

2 2

Squid: RGW overview freezes (this.dataArray[W] is undefined)
by Eugen Block 04 Feb '25

04 Feb '25

Hi, I couldn't find an existing tracker yet, so I'm asking here if this already has been reported (or even fixed). Until now, I've only been playing around with virtual test clusters on Squid, and I noticed the dashboard freezes when I'm on the /rgw/overview page. This is what I get in the mgr log: ---snip--- Jan 31 13:16:37 host2 ceph-mgr[2799]: [dashboard ERROR frontend.error] (https://IP:8443/#/rgw/overview) this.dataArray[W] is undefined updateChartData@https://IP:8443/main.5d76eec705c379ed.js:3:1175137 ngOnChanges@https://IP:8443/main.5d76eec705c379ed.js:3:1174390 ... 99140/ye/v.prototype.invoke@https://IP:8443/ ---snip--- The output is lengthy (around 100 lines), just wanted to give an impression. Clicking anywhere else doesn't really improve anything, only reloading the page lets you browse again. This seems to be related to "Multi-Site Sync Status" (which I don't have enabled), it seems to freeze while loading. But sometimes it does load successfully, showing the info ( Multi-site needs to be configured in order to see the multi-site sync status. Please consult the documentation on how to configure and enable the multi-site functionality.). But I thought, maybe that's my lab environment or something. Last week I upgraded a customer cluster to 19.2.0 as well, and it shows the same behavior. Should I create a tracker issue? Thanks! Eugen

3 3

[ceph-ansible][radosgw]: s3 key regenration issue
by Danish Khan 04 Feb '25

04 Feb '25

Dear All, I am getting 500 internal server error after deleting and creating new access and secret keys. I am able to access my buckets and and its data using new keys from S3cmd utility. but it seems ceph MGR keep cache and throwing 500 internal server error I am trying to get meta data from CEPH dashboard. But when I do ceph mgr fail and move the ceph dashboard to another MGR then it starts working fine again. Is there a way I can clear cache of MGR or is the any bug in ceph for regeneration of S3 access keys? Regards, Danish

2 4

Identify who is doing what on an osd
by Szabo, Istvan (Agoda) 04 Feb '25

04 Feb '25

Hi, Is there a way to pull some meaningful information out from the dump_ops_in_flight dump? I'd like to know which bucket this operation relates to. "description": "osd_op(client.3685135831.0:1525226003 26.144 26.4e892144 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e738686)", "initiated_at": "2025-02-04T10:35:10.009003+0700", "age": 0.018950328999999998, "duration": 0.022382503000000002, "type_data": { "flag_point": "queued for pg", "client_info": { "client": "client.3685135831", "client_addr": "10.118.199.10:0/3348448578", "tid": 1525226003 }, "events": [ { "event": "initiated", "time": "2025-02-04T10:35:10.009003+0700", "duration": 0 }, { "event": "throttled", "time": "2025-02-04T10:35:10.009003+0700", "duration": 0 }, Thank you

1 0