- ceph-users - lists.ceph.io

Orphaned CephFS objects
by Trey Palmer 08 Oct '25

08 Oct '25

Hi, Some months ago we deleted about 1.4PB net of CephFS data. Approximately 600TB net of the space we expected to reclaim, did not get reclaimed. The cluster in question is now on Reef, but it was on Pacific when this happened. The cluster has 660 15TB NVMe OSD's on 55 nodes, plus 5 separate mon nodes. We almost exclusively use it for CephFS, though there is a small RBD pool for VM images. At the time we had a single MDS for the entire cluster, but we are now breaking that CephFS instance apart into multiple instances to spread the load (we tried multiple MDS ranks but it regressed performance and caused a near-disaster). Anyway, we often delete ~500TB at a time and we haven't previously run into this orphaned object problem. At least, not at a scale big enough for us to notice -- generally I've gotten back the space I expected to free up. Also we have a 30PB raw spinning disk cluster running Quincy that we deleted 2 PB net from at the same time, and it didn't exhibit this problem. So my assumption is that there is a bug in the version we were running that we exposed by deleting so much data at once. Our support vendor did some digging using some special utilities and found that there are a bunch of orphaned objects from 2 particular snapshots, where the MDS has no reference to the objects, but they still exist on disk. They feel that it would be somewhat difficult and risky to try to find and delete these objects via rados. However, obviously, 600+ *net* TB of all-NVMe storage is quite a lot of money to just let go to waste. That's effectively over 1.1 PB, once you figure EC overhead and the need to not fill the cluster over about 70%. So to reclaim the pool I have created a plan to move everything off of the pool, and then delete the pool. Which we are largely needing to do anyway, due to splitting to multiple CephFS instances. My question is, am I likely to run into problems with this? For example, will I be able to do 'ceph fs rm_data_pool' once there are no longer any objects associated with the CephFS instance on the pool, or will the MDS have ghost object records that cause the command to balk? Thanks a lot for any insight, Trey Palmer

7 15

v20.1.1 Tentacle RC1 released
by Yuri Weinstein 07 Oct '25

07 Oct '25

This is the second release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 and Rocky 10 were built for this RC0 Feature highlights: * RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. There is a new implementation of the Erasure Coding I/O code which provides substantial performance improvements and some capacity improvements. The new code is designed to optimize performance when using Erasure Coding with block storage (RBD) and file storage (CephFS) but will have some benefits for object (RGW) storage, in particular when using smaller sized objects. A new flag allow_ec_optimizations needs to be set on each pool to switch to using the new code, existing pools can be upgraded once the OSD and MON daemons have been updated; there is no need to update the clients. The default erasure code plugin has been switched from Jerasure to ISA-L. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. Data Availability Score: Users can now track data availability score for each pool in their cluster. This feature is currently in tech preview.A pool is considered unavailable if any PG in the pool is not in active state or if there are unfound objects. Otherwise the pool is considered available. The score is updated every one second by default. The feature is on by default. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. * Integrated SMB support Ceph clusters now support an smb manager module that works similarly to the existing nfs subsystem. The new smb support allows the ceph cluster to automatically create Samba-backed SMB file shares connected to CephFS. The smb module can configure both basic Active Directory domain or standalone user authentication. The Ceph cluster can host one or more virtual smb cluster which can be truly clustered using Samba's CTDB technology. The smb module requires a cephadm enabled Ceph cluster and deploys container images provided by the samba-container project. The Ceph dashboard can be used to configure smb clusters and shares. A new cephfs-proxy daemon is automatically deployed to improve the scalability and memory usage when connecting Samba to CephFS. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: dd9c546413d50a90668289255a256022ea21f0c0

1 0

Follow-up on Bug #68060: _deferred_replay can overwrite BlueFS
by Yuma Ogami 07 Oct '25

07 Oct '25

Hi, I'm writing to follow up on Bug #68060, "_deferred_replay can overwrite BlueFS". Link to the issue: https://tracker.ceph.com/issues/68060 This issue is marked as high priority and has the potential for major data corruption. I'm closely watching its progress. About a month ago, I posted a couple of questions in #note-26 on the tracker, but haven't received a response yet. I understand everyone is very busy, but I would be grateful for any updates on this matter. For convenience, I'm re-posting my questions here: Q1: The following my understanding correct? - If object data is corrupted: If there are other healthy replicas, the corruption will be detected and fixed on scrub or read. - If BlueFS is corrupted: The OSD would crash during operation or fail to start at next BlueFS's open process. Q2: And is there a new fix being developed after the following PR was reverted? https://github.com/ceph/ceph/pull/60753 Any information you could provide would be greatly appreciated. Thank you for your time and help. Best regards, Yuma Ogami

1 0

[RGW] radosgw-admin object stat is very slow on some object
by Sa Pham 06 Oct '25

06 Oct '25

Hi all, I'm running CEPH reef with 2 RGW instances. I have an issue with some object response time being very slow. I checked with command radosgw-admin object stat --bucket <bucket> --object <object> and it responses slow as well. How should I troubleshoot this issue? Thanks & Regards

2 2

Logging recommendation on docker/cephadm pacific
by J-P Methot 06 Oct '25

06 Oct '25

Hi, We've noticed that on some of our older setups (Pacific here), we're getting what appears to be duplicate logs for our ceph docker containers deployed with cephadm. These logs are located at "/var/lib/docker/containers/*containerid*-json.log", are generated by Docker and don't rotate by default. We're wondering what would be the best practice here. Would it be best to setup log rotation for these files or is it safe to just stop Docker from logging at all? -- Mes salutations distinguées | Best regards Jean-Philippe MÉTHOT (Il, Lui | He/Him) Administrateur Infra sénior | Senior Infra administrator PlanetHoster AVIS: Ce courriel est confidentiel et peut être l'objet de secret professionnel. Tout usage ou diffusion de celui-ci par une personne autre que son destinataire est illégal. Si vous recevez ce courriel par erreur, veuillez nous en aviser sans délai et le supprimer de votre système. This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake, please let us know by e-mail reply and delete it from your system.

2 1

ATTN: Potential data loss from 'radosgw-admin bucket rm --bypass-gc'
by Casey Bodley 03 Oct '25

03 Oct '25

A long-standing data loss bug with --bypass-gc was reported today in https://tracker.ceph.com/issues/73348. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey. You can follow https://tracker.ceph.com/issues/73348 to track the fix and its backports. In the meantime, use 'radosgw-admin bucket rm' without --bypass-gc which correctly handles copied objects.

1 0

help OSD deploying via yaml
by GLE, Vivien 03 Oct '25

03 Oct '25

Hi, For testing purpose we need to deploy : - 1 pool of 6 SSD OSD - 1 pool of 6 HDD OSD - 1 pool of 6 HDD OSD with 2 SSD for DB+WAL I tried to orch apply this yaml but it doesn't work as expected Osd part of the yaml -> service_type: osd service_id: osd_spec placement: hosts: - host1 - host2 - host3 - host4 - host5 - host6 data_devices: paths: - /dev/sda - /dev/sdc spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: osd_spec_wall placement: hosts: - host1 - host2 - host3 - host4 - host5 - host6 spec: data_devices: paths: - /dev/sdf db_devices: paths: - /dev/sde limit: 2 db_slots: 3 Only 1 db on /dev/sde from host1 has been created and this OSD showed up as half full at his creation: ceph osd df | grep 25 25 hdd 0.63669 1.00000 652 GiB 373 GiB 1.5 MiB 1 KiB 38 MiB 279 GiB 57.15 35.93 0 up ceph-volume lvm list ====== osd.25 ====== [block] /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block device /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block uuid vqJGf8-V5g0-S1cA-BAcN-Qm9D-9VTx-xsc8wk cephx lockbox secret cluster fsid id cluster name ceph crush device class db device /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 db uuid Rtj5KA-Qxjk-IFmY-3ffQ-gSAu-Snte-uku8HC encrypted 0 osd fsid 2f009760-fc2b-46d5-984d-e8200dfd9d9d osd id 25 osdspec affinity osd_spec_wall type block vdo 0 with tpm 0 devices /dev/sdf [db] /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 block device /dev/ceph-23d9297a-d0e1-47be-8650-5c8ccae4fe0e/osd-block-2f009760-fc2b-46d5-984d-e8200dfd9d9d block uuid vqJGf8-V5g0-S1cA-BAcN-Qm9D-9VTx-xsc8wk cephx lockbox secret cluster fsid id cluster name ceph crush device class db device /dev/ceph-9c39b87c-a39c-413f-b1ef-07881195fcb8/osd-db-feee5095-b5e7-47a0-ae87-8e5039512661 db uuid Rtj5KA-Qxjk-IFmY-3ffQ-gSAu-Snte-uku8HC encrypted 0 osd fsid 2f009760-fc2b-46d5-984d-e8200dfd9d9d osd id 25 osdspec affinity osd_spec_wall type db vdo 0 with tpm 0 devices /dev/sde The other /dev/sde device did show up as data_device instead of db_device (example here on host2) : ceph-volume lvm list ====== osd.17 ====== [block] /dev/ceph-629f98b0-5ed4-4e75-81b9-e85ca76afb15/osd-block-5d43d683-1f7f-4dc1-935e-6a79745252f9 block device /dev/ceph-629f98b0-5ed4-4e75-81b9-e85ca76afb15/osd-block-5d43d683-1f7f-4dc1-935e-6a79745252f9 block uuid HQpp1l-x7IB-kA2W-6gWO-BGlM-VN2k-vYf43R cephx lockbox secret cluster fsid id cluster name ceph crush device class encrypted 0 osd fsid 5d43d683-1f7f-4dc1-935e-6a79745252f9 osd id 17 osdspec affinity osd_spec type block vdo 0 with tpm 0 devices /dev/sde Thx for your help Vivien

3 8

tentacle 20.1.1 RC1 QE validation status
by Yuri Weinstein 03 Oct '25

03 Oct '25

Details of this release are summarized here: https://tracker.ceph.com/issues/73123#note-1 Release Notes - TBD LRC upgrade - TBD Dev Leads, please review the list of suites for completeness, as this is a new release. Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Adam King & Guillaume rbd: Ilya krbd: Ilya nvmeof: Aviv upgrade/squid-x: Laura upgrade/reef-x: Laura upgrade/telemetry-upgrade: PASSED upgrade/tentacle-p2p: N/A upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED perf-basic: PASSED crimson-rados: PASSED powercycle: PASSED TIA

8 13

Re: Ceph PG stuck at degraded and very slow
by Sa Pham 03 Oct '25

03 Oct '25

Hi Anthony, I dont see any error from dmesg log and smart of those OSDs Regards On Fri, 3 Oct 2025 at 07:08 Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Check dmesg / var/log/messages on the hosts with that PG’s OSDs for > errors, and run a `smartctl -a` on each of those OSDs’ drives. > > > On Oct 2, 2025, at 7:55 PM, Sa Pham <saphi070(a)gmail.com> wrote: > > > > Hi Joshua, > > > > No. OSD is still responding. > > > > # ceph tell osd.130 version > > { > > "version": "18.2.7-0-g6b0e988052e", > > "release": "reef", > > "release_type": "stable" > > } > > > > But the primary OSD which includes slow pg (18.773) will respond slower. > > > > > > detailed as below > > > > > > # ceph pg dump_stuck degraded > > PG_STAT STATE UP > > UP_PRIMARY ACTING ACTING_PRIMARY > > 18.773 active+undersized+degraded+remapped+backfilling [302,150,138] > > 302 [130,101] 130 > > > > > > # time ceph tell osd.130 version > > { > > "version": "18.2.7-0-g6b0e988052e", > > "release": "reef", > > "release_type": "stable" > > } > > > > real 0m2.113s > > user 0m0.148s > > sys 0m0.036s > > # time ceph tell osd.101 version > > { > > "version": "18.2.7-0-g6b0e988052e", > > "release": "reef", > > "release_type": "stable" > > } > > > > real 0m0.192s > > user 0m0.152s > > sys 0m0.037s > > > > > > I don't know why. > > > > > > Regards, > > > > > > > > On Fri, Oct 3, 2025 at 4:21 AM Joshua Blanch <joshua.blanch(a)clyso.com> > > wrote: > > > >> could it be an OSD not responding? > >> > >> I would usually do > >> > >> ceph tell osd.* version > >> > >> To test if you can connect with osds > >> > >> > >> On Thu, Oct 2, 2025 at 12:09 PM Sa Pham <saphi070(a)gmail.com> wrote: > >> > >>> *Hello everyone,* > >>> > >>> I’m running a Ceph cluster used as an RGW backend, and I’m facing an > issue > >>> with one particular placement group (PG). > >>> > >>> > >>> - > >>> > >>> Accessing objects from this PG is *extremely slow*. > >>> - > >>> > >>> Even running ceph pg <pg_id> takes a very long time. > >>> - > >>> > >>> The PG is currently *stuck in a degraded state*, so I’m unable to > move > >>> it to other OSDs. > >>> > >>> > >>> Current ceph version is reef 18.2.7. > >>> > >>> Has anyone encountered a similar issue before or have any suggestions > on > >>> how to troubleshoot and resolve it? > >>> > >>> > >>> Thanks in advance! > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > >> > > > > -- > > Sa Pham Dang > > Skype: great_bn > > Phone/Telegram: 0986.849.582 > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1