- ceph-users - lists.ceph.io

Experience with 100G Ceph in Proxmox
by Giovanna Ratini 11 Apr '25

11 Apr '25

Hello everyone, We are running Ceph in Proxmox with a 10G network. Unfortunately, we are experiencing very low read rates. I will try to implement the solution recommended in the Proxmox forum. However, even 80 MB per second with an NVMe drive is quite disappointing. Forum link <https://forum.proxmox.com/threads/slow-performance-on-ceph-per-vm.151223/#p…> For this reason, we are considering purchasing a 100G switch for our servers. This raises some questions: Should I still use separate networks for VMs and Ceph with 100G? I have read that running Ceph on bridged connections is not recommended. Does anyone have experience with 100G Ceph in Proxmox? Is upgrading to 100G a good idea, or will I have 60G sitting idle? Thanks in advance! Gio

13 32

v19.2.2 Squid released
by Yuri Weinstein 11 Apr '25

11 Apr '25

We're happy to announce the 2nd backport release in the Squid series. https://ceph.io/en/news/blog/2025/v19-2-2-squid-released/ Notable Changes --------------- - This hotfix release resolves an RGW data loss bug when CopyObject is used to copy an object onto itself. S3 clients typically do this when they want to change the metadata of an existing object. Due to a regression caused by an earlier fix for https://tracker.ceph.com/issues/66286, any tail objects associated with such objects are erroneously marked for garbage collection. RGW deployments on Squid are encouraged to upgrade as soon as possible to minimize the damage. The experimental rgw-gap-list tool can help to identify damaged objects. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.2.2.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 0eceb0defba60152a8182f7bd87d164b639885b8

4 3

Cannot reinstate ceph fs mirror because i destroyed the ceph fs mirror peer/ target server
by Jan Zeinstra 11 Apr '25

11 Apr '25

Hi, This is my first post to the forum and I don't know if it's appropriate, but I'd like to express my gratitude to all people working hard on ceph because I think it's a fantastic piece of software. The problem I'm having is caused by me; we had a well working ceph fs mirror solution; let's call it source cluster A, and target cluster B. Source cluster A is a modest cluster consisting of 6 instances, 3 OSD instances, and 3 mon instances. The OSD instances all have 3 disks (HDD's) and 3 OSD demons, totalling 9 OSD daemons and 9 HDD's. Target cluster B is a single node system having 3 OSD daemons and 3 HDD's. Both clusters run ceph 18.2.4 reef. Both clusters use Ubuntu 22.04 as OS throughout. Both systems are installed using cephadm. I have destroyed cluster B, and have built it from the ground up (I made a mistake in PG sizing in the original cluster) Now i find i cannot create/ reinstate the mirroring between 2 ceph fs filesystems, and i suspect there is a peer left behind in the filesystem of the source, pointing to the now non-existent target cluster. When i do 'ceph fs snapshot mirror peer_list prodfs', i get: '{"f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5": {"client_name": "client.mirror_remote", "site_name": "bk-site", "fs_name": "prodfs"}}' When i try to delete it: 'ceph fs snapshot mirror peer_remove prodfs f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5', i get: 'Error EACCES: failed to remove peeraccess denied: does your client key have mgr caps? See http://docs.ceph.com/en/latest/mgr/administrator/#client-authentication', but the logging of the daemon points to the more likely reason of failure: ---- Apr 08 12:54:26 s1mon systemd[1]: Started Ceph cephfs-mirror.s1mon.lvlkwp for d0ea284a-8a16-11ee-9232-5934f0f00ec2. Apr 08 12:54:26 s1mon cephfs-mirror[310088]: set uid:gid to 167:167 (ceph:ceph) Apr 08 12:54:26 s1mon cephfs-mirror[310088]: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable), process cephfs-mirror, pid 2 Apr 08 12:54:26 s1mon cephfs-mirror[310088]: pidfile_write: ignore empty --pid-file Apr 08 12:54:26 s1mon cephfs-mirror[310088]: mgrc service_daemon_register cephfs-mirror.22849497 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7a> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: remote monitor host=[v2:172.17.16.12:3300/0,v1:172.17.16.12:6789/0] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57c51ba640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::Utils connect: error connecting to bk-site: (13) Permission denied Apr 08 12:54:30 s1mon conmon[310082]: 2025-04-08T10:54:30.365+0000 7f57d81e0640 -1 cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cl> Apr 08 12:54:30 s1mon cephfs-mirror[310088]: cephfs::mirror::PeerReplayer(f3ea4e15-6d77-4f28-aacb-9afbfe8cc1c5) init: error connecting to remote cluster: (13) Permission denied Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.362+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.386+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.430+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 09 00:00:16 s1mon conmon[310082]: 2025-04-08T22:00:16.466+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 09 00:00:16 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.767+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.811+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.851+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon conmon[310082]: 2025-04-09T22:00:01.891+0000 7f57d99e3640 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm()> Apr 10 00:00:01 s1mon cephfs-mirror[310088]: received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 ---- Is there any chance I can get the mirroring daemon to forget about the cluster I lost ? Best regards, Jan Zeinstra

2 1

endless remapping after increasing number of PG in a pool
by Michel Jouvin 11 Apr '25

11 Apr '25

Hi, We are observing a new strange behaviour on our production cluster : we increased the number of PG (from 256 to 2048) in a (EC) pool after a warning that there was a very high number of objects per pool (the pool has 52M objects). Background: this happens in the cluster that had a strange problem last week, discussed in the thread "Production cluster in bad shape after several OSD crashes". The PG increase was done after the cluster returned to a normal state. The increase of the number of PG resulted in 20% misplaced objects and ~160 PG remapped (over 256). As there is no much user activity on this cluster (except on this pool) these days, we decided to set the mclock profile to high_recovery_ops. We also disabled autoscaler on this pool (it was enabled and it is not clear why we add the warning with autoscaler enabled). The pool was created with --bulk. The remapping went steadily during 2-3 days (as much as we can tell) but when reaching the end (between 0.5% and 1% of misplaced objects, ~10 PG remapped) it readds remapped PG (that can be seen with 'ceph pg dump_stuck'), all belonging to the pool affected by the increase. This already happened 3-4 times and it is very unclear why. There was no specific problem reported on the cluster that may explain this (no OSD down). I was wondering if the balancer may be responsible for this but I don't have the feeling it is the case: first the balancer doesn't report about doing something (but I may miss the history), second the balancer would probably affect PG from different pools. (there are 50 ppols in the cluster). There are 2 warnings that may or may not be related: - All mon have their data size > 15 GB (mon_data_size_warn). Currently 16 GB. It happened progressively during the remapping on all 3 mon, I guess it is due to the operation in progress and is harmless. Do you confirm? - Since yesterday we have 4 PG that have not been deep scrubbed in time, belonging to different pools. Again, I tend to correlate this to the remapping in progress putting too much load (or other constraints), as there are a lot of deep scrubs per day. The current age distribution of deep scrubs is: 4 "2025-03-19 21 "2025-03-20 46 "2025-03-21 35 "2025-03-22 81 "2025-03-23 597 "2025-03-24 1446 "2025-03-25 2234 "2025-03-26 2256 "2025-03-27 1625 "2025-03-28 1980 "2025-03-29 2993 "2025-03-30 3871 "2025-03-31 1113 "2025-04-01 Should we worry about the situation? If yes, what would you advise to look at or do? To clear the problem last week, we had to restart all OSD but we didn't restart mon. Do they play a role in deciding the remapping plan? Is restarting them something that may help? As usual, thanks in advance for any help/hint. Best regards, Michel

2 4

Image Live-Migration does not respond to OpenStack Glance images
by Yuta Kambe (Fujitsu) 10 Apr '25

10 Apr '25

Hi everyone. I am trying Image Live-Migration but it is not working well and I would like some advice. https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ I use Ceph as a backend for OpenStack Glance. I tried to migrate the Pool of Ceph used in Glance to the new Pool. Source Pool: - images_meta : metadata pool, Replication - images_data : data pool, Erasure Code Target Pool: - images_meta: metadata pool, Replication (Same as source Pool) - images_data_hdd: data pool, Erasure Code The following command I executed, but did not return a response. rbd migration prepare images_meta/image_name images_meta/image_name --data-pool images_data_hdd I checked the logs in /var/log/messages and /var/log/ceph, but no useful information was available. I would like some advice on this. - Are there any other logs I should check? - Is there a case where the rbd migration command cannot be executed? The following is supplemental information. - ceph version 17.2.8 - The migration of the OpenStack Nova image was successful with the same Pool configuration and command. - I don't know if it is related, but there is a snapshot in the image of Glance, and unprotect of the snapshot is also unresponsive. rbd snap unprotect images_meta/image_name@snap

2 1

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

NIH Datasets
by Linas Vepstas 09 Apr '25

09 Apr '25

OK what you will read below might sound insane but I am obliged to ask. There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere. I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know. While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list. OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment. Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope? Am I misunderstanding something about IPFS that merits taking a second look at it? Is there any other way of getting scalable reliable "decentralized" internet-wide storage? I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work? -- Linas -- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.

9 17

Diskprediction_local mgr module removal - Call for feedback
by Yaarit Hatuka 09 Apr '25

09 Apr '25

Hi everyone, On today's Ceph Steering Committee call we discussed the idea of removing the diskprediction_local mgr module, as the current prediction model is obsolete and not maintained. We would like to gather feedback from the community about the usage of this module, and find out if anyone is interested in maintaining it. Thanks, Yaarit

7 12

RBD Block alignment 16k for Databases
by filip Mutterer 09 Apr '25

09 Apr '25

Hi! Is Block alignment for Databases a thing with RBD? I heard something with 16K alignment ... If yes how can be checked what alignment a RBD got. And yes I need to run my Database in Ceph as I have no other storage with redundandency. Hints for optimizing a single Block device for a Database are very welcome. Greetings filip

2 1

Use cephfs when cephfs-data-scan cleanup is not finished
by Jiří Župka 09 Apr '25

09 Apr '25

Dear conference, I would like to ask if is secure use cephfs when cephfs-data-scan cleanup is not finished yet? Best regards, Jiri

2 1