- ceph-users - lists.ceph.io

Scub VS Deep Scrub
by Alex 20 Aug '25

20 Aug '25

I'm trying to understand which settings control light scrubbing and which control deep scrubs. `osd_scrub_begin_hour` and ` osd_scrub_end_hour` are for deep scrubs as far as I can tell even though the documentation just calls them scrubs. What about `osd_scrub_min_interval` and `osd_scrub_max_interval` for example, are they for light or deep scrubs or do these settings control both? The documentation [ https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/ ] is unclear except for around 6 cases in which "deep scrub", "deep-scrub", and "deep--scrub" are all used interchangeably.

1 0

Squid on 24.04 does not have a Release file
by Devender Singh 20 Aug '25

20 Aug '25

Dear all Getting below error, any fix for this? # apt update Hit:1 https://repo.zabbix.com/zabbix/6.4/ubuntu noble InRelease Hit:2 http://security.ubuntu.com/ubuntu noble-security InRelease Ign:3 https://download.ceph.com/debian-squid noble InRelease Err:4 https://download.ceph.com/debian-squid noble Release 404 Not Found [IP: 158.69.68.124 443] Hit:5 http://archive.ubuntu.com/ubuntu noble InRelease Hit:6 http://archive.ubuntu.com/ubuntu noble-updates InRelease Hit:7 http://archive.ubuntu.com/ubuntu noble-backports InRelease Reading package lists... Done E: The repository 'https://download.ceph.com/debian-squid noble Release' does not have a Release file. N: Updating from such a repository can't be done securely, and is therefore disabled by default. N: See apt-secure(8) manpage for repository creation and user configuration details. # cat /etc/os-release PRETTY_NAME="Ubuntu 24.04.3 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.3 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble LOGO=ubuntu-logo # cat /etc/apt/sources.list.d/ceph.list deb https://download.ceph.com/debian-squid/ noble main Regards Dev

3 4

[bluestore] How to deal with free fragmentation
by Florent Carli 20 Aug '25

20 Aug '25

Hello, I’m running Ceph 19.2.3 with cephadm on a 3-node cluster, and I recently started seeing the following warning related to free fragmentation: HEALTH_WARN 3 OSD(s) [WRN] BLUESTORE_FREE_FRAGMENTATION: 3 OSD(s) osd.0 0.802713 osd.1 0.803276 osd.2 0.803870 I haven’t been able to find clear guidance on how to address this issue. Are there any recommended approaches or best practices for reducing BlueStore free space fragmentation? Thanks in advance for your help! Florent.

4 3

Issue with Ceph RBD Incremental Backup (import-diff failure)
by Vishnu Bhaskar 20 Aug '25

20 Aug '25

I was trying to back up volumes created in Ceph using *rbd export* and *rbd import*. For incremental backups, I used *rbd export-diff*. I was able to take the full backup using *rbd export*, but when I tried to import the incremental backup using *rbd import-diff*, I encountered an error: Steps I followed to take the backup: # cd volume-ee6f1396-7082-4475-bc0e-d7005b3331db/ # rbd snap create volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap1-$(date +%Y%m%dT%H%M) # rbd export volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap1-20250820T1227 snap1-20250820T1227.rbd2 --export-format --------- Write data in to the volume # rbd snap create volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap2-$(date +%Y%m%dT%H%M) # rbd snap ls volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db SNAPID NAME SIZE PROTECTED TIMESTAMP 202 snap1-20250820T1227 1 GiB Wed Aug 20 12:27:16 2025 203 snap2-20250820T1231 1 GiB Wed Aug 20 12:31:08 2025 # rbd export-diff volumes/volume-ee6f1396-7082-4475-bc0e-d7005b3331db@snap2-20250820T1231 --from-snap snap1-20250820T1227 snap2-20250820T1231.rbdiff # ls -lh total 782M -rw-r--r-- 1 adminuser adminuser 600M Aug 20 12:30 snap1-20250820T1227.rbd2 -rw-r--r-- 1 adminuser adminuser 182M Aug 20 12:37 snap2-20250820T1231.rbdiff # rbd import snap1-20250820T1227.rbd2 volumes/vm1 Importing image: 100% complete...done. # rbd import-diff snap2-20250820T1231.rbdiff volumes/vm1 start snapshot 'snap1-20250820T1227' does not exist in the image, aborting Importing image diff: 0% complete...failed. rbd: import-diff failed: (22) Invalid argument Thanks and Regards Vishnu Bhaskar Acceleron Labs Pvt Ltd Bangalore, India

1 0

MAX_AVAIL becomes 0 bytes when setting osd crush weight to low value.
by Justin Mammarella 19 Aug '25

19 Aug '25

Hello, We’re seeing the MAX_AVAIL value in ceph df instantaneously drop to 0 Bytes / 100% full when specific osds have their crush weight set to low values. The osds are otherwise healthy, and ceph osd df does not show their utilization to be above 70%. ceph version 19.2.2 CLASS SIZE AVAIL USED RAW USED %RAW USED mf1hdd 19 PiB 8.9 PiB 10 PiB 10 PiB 53.83 to POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL mf1fs_data 1 16384 6.8 PiB 2.51G 10 PiB 100.00 0 B We’re running a 9+3 EC pool. This cluster has 1139 osds / 46 host. We’re in the process of downsizing the cluster and draining nodes via crush reweight is part of our normal operations. It happened once a few weeks ago, and we isolated the issue to the weight on a single osd. Now it’s happening during rebalance on multiple osds, at some point the movement of PGs triggers an edge case causing the MAX AVAIL calculation to fail if the crush weight is too low. Example crush weights 1418 mf1hdd 0.02000 osd.1418 up 1.00000 1.00000 1419 mf1hdd 0.02000 osd.1419 up 0 1.00000 2110 mf1hdd 0.02000 osd.2110 up 1.00000 1.00000 2111 mf1hdd 0.02000 osd.2111 up 0 1.00000 2112 nvmemeta 0.02000 osd.2112 up 1.00000 1.00000 Any ideas before I file a bug report? Thankyou

2 4

/var/lib/ceph/crash/posted does not exist
by lejeczek 19 Aug '25

19 Aug '25

Hi guys. I'm missing: /var/lib/ceph/crash/posted on each node, one node where 'mon' fails: -> $ ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum podster3,podster2 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon mon.podster1.mine.priv on podster1.mine.priv is in error state [WRN] MON_DOWN: 1/3 mons down, quorum podster3,podster2 mon.podster1.mine.priv (rank 0) addr [v2:10.1.1.61:3300/0,v1:10.1.1.61:6789/0] is down (out of quorum) started logging: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create Is this expected - on newly deployed cluster with 'cephadm' - that this path does not exist, with 'crash' module already enabled? Is this the SOP that user creates (also perhaps do other necessary) this path manually in order for 'crash' to be functional? many thanks, L.

3 5

Re: Follow-up on Ceph RGW Account-level API
by William Edwards 19 Aug '25

19 Aug '25

> Op 19 aug 2025 om 09:40 heeft Pavithraa AG <pavithraa.ag(a)zybisys.com> het volgende geschreven: > > Hi ceph community, > Are you, by any chance, using an LLM to write your messages? It would save everyone a lot of reading to strip the redundant parts. > > Thank you for sharing the link to the AdminOps API documentation [1]. We have reviewed it thoroughly; however, we could not find endpoints specifically related to account-level operations. > > In particular, we are looking for REST API equivalents for the following actions, so that they can be used safely in production without relying on radosgw-admin commands: > > Account creation > > > Account quota update > > > Account user management (add/remove/list users under an account) > > > Retrieve account details > > > > Could you please confirm: > > Whether there are existing REST API endpoints for these actions (possibly under a different section of the documentation)? > > > If not, whether there is a roadmap or recommended alternative for implementing these account-level operations in a production environment? > > > > Any additional guidance, examples, or references from the community would be greatly appreciated. > > > > > > > > > > > > > > > > Pavithraa AG > > Associate Software Engineer > > mailto:pavithraa.ag@zybisys.com > > https://www.zybisys.com > > > > > > https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys > https://www.facebook.com/Zybisys/ > > > > > > > > > > > > > > > > > > > > > > > > ---- On Mon, 11 Aug 2025 12:48:13 +0530 Dhivya G < mailto:dhivya.g@zybisys.com > wrote --- > > > > Dear Ceph Community, > > I hope this message finds you well. > > We are currently working on integrating Ceph RGW into our system and have encountered a challenge regarding account-level management via API. > > At present, we are using radosgw-admin commands programmatically within our code to manage and retrieve account-level information. While this approach works, we understand that relying on radosgw-admin is not recommended for production environments due to its direct interaction with the backend and potential impact on performance and stability. > > We have explored the available documentation and interfaces but could not find a dedicated Account-level API (e.g., for managing users, retrieving account metadata, etc.) that can be safely used in production over RESTful protocols (such as S3 or Swift-compatible APIs). > > We are seeking clarification on the following points: > > Is there an officially supported Account API in Ceph RGW that allows account-level operations via REST or other production-safe interfaces? > > > If not, is there a roadmap or recommended alternative for performing these actions in a production-ready manner? > > > Are there any best practices or tools from the community that can bridge this gap? > > > > We would greatly appreciate any guidance, documentation, or recommendations you can share . > > > > > > > > Best regards > > > > Dhivya G > > > > > > > > > > > https://www.facebook.com/Zybisys/ > Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. > > > > E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email." > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Follow-up on Ceph RGW Account-level API
by Pavithraa AG 19 Aug '25

19 Aug '25

Hi ceph community, Thank you for sharing the link to the AdminOps API documentation [1]. We have reviewed it thoroughly; however, we could not find endpoints specifically related to account-level operations. In particular, we are looking for REST API equivalents for the following actions, so that they can be used safely in production without relying on radosgw-admin commands: Account creation Account quota update Account user management (add/remove/list users under an account) Retrieve account details Could you please confirm: Whether there are existing REST API endpoints for these actions (possibly under a different section of the documentation)? If not, whether there is a roadmap or recommended alternative for implementing these account-level operations in a production environment? Any additional guidance, examples, or references from the community would be greatly appreciated. Pavithraa AG Associate Software Engineer mailto:pavithraa.ag@zybisys.com https://www.zybisys.com https://in.linkedin.com/company/zybisys https://www.facebook.com/Zybisys/ https://www.instagram.com/ZyBiSys/ https://www.youtube.com/@zybisys https://www.facebook.com/Zybisys/ ---- On Mon, 11 Aug 2025 12:48:13 +0530 Dhivya G < mailto:dhivya.g@zybisys.com > wrote --- Dear Ceph Community, I hope this message finds you well. We are currently working on integrating Ceph RGW into our system and have encountered a challenge regarding account-level management via API. At present, we are using radosgw-admin commands programmatically within our code to manage and retrieve account-level information. While this approach works, we understand that relying on radosgw-admin is not recommended for production environments due to its direct interaction with the backend and potential impact on performance and stability. We have explored the available documentation and interfaces but could not find a dedicated Account-level API (e.g., for managing users, retrieving account metadata, etc.) that can be safely used in production over RESTful protocols (such as S3 or Swift-compatible APIs). We are seeking clarification on the following points: Is there an officially supported Account API in Ceph RGW that allows account-level operations via REST or other production-safe interfaces? If not, is there a roadmap or recommended alternative for performing these actions in a production-ready manner? Are there any best practices or tools from the community that can bridge this gap? We would greatly appreciate any guidance, documentation, or recommendations you can share . Best regards Dhivya G https://www.facebook.com/Zybisys/ Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

2 1

Re: v19.2.3 Squid released
by Justin Owen 18 Aug '25

18 Aug '25

Hello, Is there any update on this CentOS Stream vs Enterprise Linux dependency issue? Wondering if there is any sort of workaround or a bug that I can follow for the progress. We're using OL9 as our container host, but can cannot update the ceph-common tools or ceph-volume packages (which have some bug fixes we'd like) to match the 19.2.3 of the containers. Thanks Justin

1 0

crash - auth: unable to find a keyring ... (2) No such file or directory
by lejeczek 18 Aug '25

18 Aug '25

Hi guys. I have one crash - containerized cluster - which upon start logs: ... INFO:ceph-crash:pinging cluster to exercise our key 2025-08-18T14:59:59.663+0000 7fe9106bb640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-18T14:59:59.663+0000 7fe9106bb640 -1 AuthRegistry(0x7fe908069670) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-18T14:59:59.664+0000 7fe9106bb640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2025-08-18T14:59:59.664+0000 7fe9106bb640 -1 AuthRegistry(0x7fe9106b9ff0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2025-08-18T14:59:59.665+0000 7fe90e430640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2025-08-18T14:59:59.665+0000 7fe90dc2f640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] In that 'crash' container I have: -> $ ll /etc/ceph/ total 16 -rw-------. 1 ceph ceph 81 Aug 6 13:26 ceph.client.crash.podster1.keyring -rw-------. 1 ceph ceph 253 Aug 6 13:26 ceph.conf -rw-------. 1 root root 151 Aug 18 04:20 ceph.keyring -rw-r--r--. 1 root root 92 Jul 17 03:53 rbdmap then: -> $ ceph auth get client.admin [client.admin] key = AQD.... caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" which key is in: /etc/ceph/ceph.keyring on this 'crash' What is happening here can anybody suggest/tell? Other 'crashes' remain that-error-free - but I wonder if that is only because they have had nothing to do yet. This one 'crash' has - on the same node - monitor failing. many thanks, L.

1 0