- ceph-users - lists.ceph.io

BLUEFS_SPILLOVER after Reef upgrade
by Matthew Hutchinson 24 Nov '25

24 Nov '25

Hello everyone, I hope you are doing well. My cluster is in a similar situation to the one described in this email thread: <https://www.mail-archive.com/search?l=ceph-users@ceph.io&q=subject:%22%5C%5 Bceph%5C-users%5C%5D+Re%5C%3A+BLUEFS_SPILLOVER+after+Reef+upgrade%22&o=newes t> https://www.mail-archive.com/search?l=ceph-users@ceph.io&q=subject:%22%5C%5B ceph%5C-users%5C%5D+Re%5C%3A+BLUEFS_SPILLOVER+after+Reef+upgrade%22&o=newest This cluster was originally deployed on Octopus and has since been upgraded to the latest version of Reef. We are now seeing the same BLUEFS_SPILLOVER behavior described in that thread. I was wondering if anyone else has encountered this issue and managed to resolve it without having to rebuild the OSDs. Thank you in advance for any guidance or suggestions you can share. Matthew Hutchinson Solution Specialist 1-866-594-7199 ext. 818 <mailto:mhutchinson@45drives.com> mhutchinson(a)45drives.com <http://www.45drives.com/> www.45drives.com

1 0

CLI pool creation with application
by Eugen Block 24 Nov '25

24 Nov '25

Hi, I do most of my Ceph stuff on the command line, although the dashboard can be quite helpful with a lot of stuff, too. But when I create a new pool on a newly deployed cluster I always have to enable the application manually. And since many of those systems are just for testing something quickly (and maybe throw them away afterwards), they aren't attached to any sort of automation (virtual machines in almost air-gapped environments). So I was wondering if there's a chance to add a CLI option to directly set the application during pool creation. The dashboard has that option, but in the background it also just sets the application after pool creation: log [INF] : from='mgr.5307102 192.168.168.50:0/604015211' entity='mgr.host1.vhylmr' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "test-appl", "app": "rbd", "yes_i_really_mean_it": true}]': finished Is this something I could add to my wish list? :-) Thanks, Eugen

1 0

Meaning of RGW sync status "full sync: objects completed"
by Adam Prycki 24 Nov '25

24 Nov '25

Hello, I have a question about rgw synchronization mechanic. When we are trying to initialize full bucket sync on existing bucket, or when try to synchonize delete bucket we see "full sync" in bucket sync status on our clusters. For example: realm b6ca44da-921e-42da-b377-80e9fae83b61 (s3-TEST.kmd4.pl) zonegroup c41b36c6-2031-407b-bab4-36cd0b3e56da (poznan-1) zone 91a17d0c-84a0-4706-a40d-1b6df1996a10 (zone-poznan-1-kielman) bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) current time 2025-11-24T13:11:48Z source zone 8e5c54a9-7eb2-4521-b327-081a1a647205 (zone-poznan-1-pcss-bst) source bucket :1d2848a2-9ca7-402c-9976-a6dc775547bb[8e5c54a9-7eb2-4521-b327-081a1a647205.4079816.2]) full sync: 15221490 objects completed In this case we have deleted bucket with broken synchronization on the secondary zone. After RGW was started up again synchronization should have tarted from zero. Bucket 1d2848a2-9ca7-402c-9976-a6dc775547bb has 1163263 10MiB s3 objects. bucket sync status is right now at 15221490 and still increasing. That's 13x bigger number than object count in this bucket. In case of some other bucket we saw that "full sync: objects completed" number grew to about 20x s3 object count in the bucket. Can someone explain what this number means and why it gets so big? How big it's supposed to get? Can we predict how much of the synchronization is "done" from this number? Best regards Adam Prycki

1 0

wrong lvs attribut
by Albert Shih 24 Nov '25

24 Nov '25

Hi everyone, I got a very strange issue, after a reboot the lvs attribut of a disk are missing «ao» osd-block-2d30e765-d02e-4dd1-9eee-16c67e539263 ceph-1efb4c4d-f7ac-4511-a593-20db66acaf0d -wi-ao---- 2.91t osd-block-bab12e42-4bbd-4225-a120-bfff72990a90 ceph-24377b3a-1bcc-45f3-8f6c-e7cabb0c212d -wi------- 3.49t osd-block-13630348-cf13-4387-8476-c13ee0e14633 ceph-3c335428-129b-4787-9195-0b67eb6dfba6 -wi-ao---- 2.91t osd-block-187e637e-c235-4e36-99eb-849c30561b09 ceph-8bde9b87-167b-4da8-9447-004ecdfa6a4c -wi-ao---- 2.91t so of course the osd daemon is unable to start. It's not the first time I got this issue, it's not a big deal because (at least until now) a reboot solve the problem. But anyone have any clue why sometime after a reboot the disk (nvme ssd) are in non active and non open state ? Hardware issue ? (I can't remember if it was the same osd). Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 24 nov. 2025 13:20:53 CET

1 0

Best way to remove a node from cluster temp to replace faulty network cable
by gagan tiwari 24 Nov '25

24 Nov '25

Hi Guys, We are getting an increasing number of RX errors on the ports on some of the nodes in the cluster including mon , mds and osd nodes In an attempt to fix it , we plan to replace SFPs / cables on these nodes. I think I will need to temp remove those nodes one by one from the cluster and replace the network cables / SFPs and add them back to cluster So, plz let me know the best / safest way to proceed. Thanks, Gagan

4 4

CephFS: large omap object
by Philipp Hocke 21 Nov '25

21 Nov '25

Hi, last week we had a "large omap" warning in one of our clusters. The affected object was 025-11-09T23:31:24.799+0000 7f70131b3640 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 2:ddb71591:::mds1_openfiles.d:head PG: 2.89a8edbb (2.b) Key count: 200001 Size (bytes): 12122520 I wanted to get a deeper understanding on what triggers this specific issue and had a look at the code - I'm not that deep into C++, so please, correct me if I'm wrong. If I understand correctly, LARGE_OMAP warnings are only generated (and cleared) during deep-scrubs. If an object is found with more keys than osd_deep_scrub_large_omap_object_key_threshold, it will trigger the warning. In this case mds1_openfiles.d had a Key count of 200001 while this specific option got deep-scrubbed and that triggered the warning The MDS should create a new openfiles fragment when the threshold is reached. My question is: How exactly does the MDS end up with an openfiles segment with more than 200000? Shouldn't the MDS create a new segment as soon as the previous one reaches the configured limit? If this is considered a bug, I'm happy to open a report. Best regards Philipp -- Philipp Hocke Leiter Linux Systemadministration / Head Of TechOps CM4ALL GmbH Im Mediapark 6A - 50670 Köln / Cologne Phone +49-(0)221-6601-0 <https://outlook.office.com/mail/sentitems> Fax +49-(0)221-6601-1011 <https://outlook.office.com/mail/sentitems> E-Mail: philipp.hocke(a)cm4all.com Internet: www.cm4all.com

2 2

Rados data export tool from ERR and unfound object cluster
by Özkan Göksu 21 Nov '25

21 Nov '25

Hello folks. I got a failed cluster recovery job and the cluster is: 2PB EC 8+2 pool used as RGW S3 storage. (14.2.16 nautilus) I fixed most of the issues and rescued a lot of failed OSD's but some OSD's are not recoverable and the cluster size is huge, recovery takes time. Fixing all the issues is impossible because missing OSD's and running PG repair take weeks. I did my best and the current state is enough for starting data export. I only need to export the healthy data first and then I'm gonna destroy-rebuild the cluster. I tried to export data by using rclone and it runs good at the beginning but when the radosgw service hits an unfound object it slows down and it freezes. Because of this issue I can not export data by using rclone+radosgw I'm using these parameters for the export: > rclone copy --files-from=object-list.txt --progress --checkers=8 > --transfers=8 --no-traverse \ > --no-update-modtime --ignore-checksum --ignore-size --no-gzip-encoding > --s3-no-check-bucket \ > --buffer-size=1G --low-level-retries=1 --retries=1 --retries-sleep=100ms > --timeout=30s \ > --contimeout=30s --ignore-errors --ignore-existing > --log-file=export-rclone.log --log-level=INFO Current state is: > # ceph health detail (14.2.16 nautilus) > HEALTH_ERR 3303 large omap objects; 1460729/1238267531 objects unfound > (0.118%); Possible data damage: 2 pgs recovery_unfound, 1 pg > backfill_unfound; Degraded data redundancy: 5487055/12315635595 objects > degraded (0.045%), 7 pgs degraded, 7 pgs undersized; 1987 pgs not > deep-scrubbed in time; 1824 pgs not scrubbed in time > LARGE_OMAP_OBJECTS 3303 large omap objects > 165 large objects found in pool 'xyz.rgw.log' > 3138 large objects found in pool 'xyz.rgw.buckets.index' > Search the cluster log for 'Large omap object found' for more details. > OBJECT_UNFOUND 1460729/1238267531 objects unfound (0.118%) > pg 56.73a has 99125 unfound objects > pg 56.470 has 572028 unfound objects > pg 56.3a9 has 564371 unfound objects > pg 56.346 has 7871 unfound objects > pg 56.36 has 1 unfound objects > pg 56.1e2 has 108269 unfound objects > pg 56.204 has 109064 unfound objects > PG_DAMAGED Possible data damage: 2 pgs recovery_unfound, 1 pg > backfill_unfound > pg 56.36 is active+backfill_unfound+undersized+degraded+remapped, > acting [166,36,7,46,152,193,128,223,80,2147483647], 1 unfound > pg 56.3a9 is active+recovery_unfound+undersized+degraded+remapped, > acting [153,69,177,156,49,167,2,2147483647,2147483647,31], 564371 unfound > pg 56.470 is active+recovery_unfound+undersized+degraded+remapped, > acting [135,2147483647,2147483647,176,37,85,65,48,183,224], 572028 unfound > PG_DEGRADED Degraded data redundancy: 5487055/12315635595 objects degraded > (0.045%), 7 pgs degraded, 7 pgs undersized > pg 56.36 is stuck undersized for 47661.320358, current state > active+backfill_unfound+undersized+degraded+remapped, last acting > [166,36,7,46,152,193,128,223,80,2147483647] > pg 56.1e2 is stuck undersized for 47660.312165, current state > active+recovering+undersized+degraded+remapped, last acting > [2147483647,127,77,140,167,195,220,131,113,31] > pg 56.204 is stuck undersized for 47660.325926, current state > active+recovering+undersized+degraded+remapped, last acting > [2147483647,42,173,17,64,189,22,113,136,158] > pg 56.346 is stuck undersized for 47660.335077, current state > active+recovering+undersized+degraded+remapped, last acting > [63,108,53,140,132,2147483647,183,171,0,98] > pg 56.3a9 is stuck undersized for 4403604.202577, current state > active+recovery_unfound+undersized+degraded+remapped, last acting > [153,69,177,156,49,167,2,2147483647,2147483647,31] > pg 56.470 is stuck undersized for 47661.762140, current state > active+recovery_unfound+undersized+degraded+remapped, last acting > [135,2147483647,2147483647,176,37,85,65,48,183,224] > pg 56.73a is stuck undersized for 47656.126741, current state > active+recovering+undersized+degraded+remapped, last acting > [2147483647,189,85,155,170,106,40,25,222,121] > PG_NOT_DEEP_SCRUBBED 1987 pgs not deep-scrubbed in time I considered "bucket check" but each bucket has 10M objects and 100TB of data. It will take days-weeks and it may fail because of the cluster state. > "rgw.main": { > "size": 106443540756257, > "size_actual": 106463698989056, > "size_utilized": 106443540756257, > "size_kb": 103948770270, > "size_kb_actual": 103968456044, > "size_kb_utilized": 103948770270, > "num_objects": 9855872 > }, I'm thinking of writing a rados exporter script and I need to do this: 1. Read bucket index and have a list of objects. 2. Check the bucket data by each object and eliminate the unfound objects and generate a clean list with healthy objects 3. Export the data directly from rados to an NFS share by combining multipart data. My questions are: 1. Are there any other ways to use radosgw+rclone for fast export from ERROR state cluster? 2. If not, then do you know any rados export tool I can use? I really need your advice. What is the best approach to this? Thank you.

1 0

November Ceph Science User Group Virtual Meeting
by Kevin Hrpcek 20 Nov '25

20 Nov '25

Hey Folks, We will be having a Ceph science/research/big cluster call on Tuesday November 25th. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments, though anyone is welcome, where we discuss whatever is on our minds regarding Ceph. Updates, outages, features, maintenance, etc...there is no single set presenter for the length of the meeting, but it is somewhat moderated to keep the conversation flowing. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments, you are most welcome to contact me. We usually start each meeting with a brief 10-15 minutes talk given by a community member, but currently don't have one scheduled for this meeting.If you are interested in signing up to give a short talk for a meeting please let me know! It is a very informal talk. Virtual event details: November 25th, 2025 9am Central US 15:00 UTC 4pm Central European Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_20251125 Pad Index: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone https://meet.jit.si/ceph-science-wg Ceph Community Calendar -- https://ceph.io/en/community/meetups/ See you there! Kevin

1 0

Ceph Community Manager Roundup Email
by Anthony Middleton 20 Nov '25

20 Nov '25

Hello, Ceph Friends! I hope this week is treating you well. Here’s a quick roundup of what’s new in the Ceph community and how you can get involved! *Tentacle (v20.2.0) Is Officially Released!* Ceph Tentacle is out in the wild! A huge thank you to everyone who contributed code, testing, docs, and feedback. This release reflects the hard work and collaboration of so many community members. Congratulations to all of you! Read the full release notes and upgrade guidance: *https://t.ly/CephTentacle20.2 <https://t.ly/CephTentacle20.2>* *Cephalocon 2025 Recordings Are Live* Miss a session in Vancouver? Want to rewatch your favorite talks? All Cephalocon recordings are now available on YouTube. Watch the playlist: *https://t.ly/WatchCephalocon25 <https://t.ly/WatchCephalocon25>* *New Office Hours with Radoslaw Zarzynski* Radoslaw Zarzynski, Software Engineering Expert at IBM, is launching a new biweekly *RADOS Office Hour. *This is an informal space to collaborate on pull requests, ask questions, walk through tricky parts of the codebase, or even do some pair programming. *When:* Every other Friday, 08:00–10:00 AM PT *Where:* Jitsi *Agenda:* Show up and hack together *Calendar:* Add it here: https://calendar.google.com/calendar/b/1?cid=OXRzOWM3bHQ3dTF2aWMyaWp2dnFxbG… *Join the New Ceph Subreddit* Our previous subreddit was banned (long story…), so we’ve launched a new home on Reddit. Join us at *r/ceph_storage*: https://www.reddit.com/r/ceph_storage/ If you have experience getting subreddits reinstated, please reach out: *community-manager(a)ceph.io <community-manager(a)ceph.io>* *New Funding Request Form* We’ve replaced the old email workflow with a much simpler *Asana Funding Request Form*. Your request now goes straight to the Community Manager + Operations Team for faster review. Submit requests here: *https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717 <https://form.asana.com/?k=7aCHVRhp0x1Ga1nOCXlckQ&d=9283783873717>* (As always, not all requests can be approved, but every submission will receive a detailed response.) *We Need Your Voice: Ceph Community Engagement Survey* This 2-minute survey helps us understand what’s working and what needs improvement across the community. I worked with survey expert Laura Flores to make responses more straightforward to process. Your input matters more than ever. Take the survey: *https://t.ly/Engagement-Survey25 <https://t.ly/Engagement-Survey25>* Thanks for being part of the Ceph community and helping shape what comes next. If you ever have ideas, questions, or just want to chat, feel free to reach out to me. Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

ATTN: Regarding Tentacle packages for Ubuntu 24.04
by Casey Bodley 20 Nov '25

20 Nov '25

The recent Tentacle v20.2.0 release includes packages for Ubuntu 24.04. We've become aware of significant issues in ceph-mgr related to python 3.12 and later, tracked in https://tracker.ceph.com/issues/73930. We don't yet have test coverage for Ubuntu 24.04, so didn't catch this in time for the release. The only workaround for this issue is to deploy ceph-mgr instances as packages on Ubuntu 22.04 or CentOS Stream 9, or as containers using cephadm. For this reason, we've removed all Ubuntu 24.04 packages from 20.2.0. We plan to fully test and support Ubuntu 24.04 and Rocky 10 in a point release.

1 0