- ceph-users - lists.ceph.io

Default firewall zone
by Sake Ceph 14 Aug '25

14 Aug '25

Is it possible to set the firewall zone to use for the orchestrator/cephadm to use? If we don't provide the zone when configuring the firewall, the default zone is drop in our company (don't ask me why). Kind regards, Sake

1 0

Backup Best Practices
by Anthony Fecarotta 14 Aug '25

14 Aug '25

Hello, I'm curious how the community handles backup and disaster recovery planning with Ceph in production environments. Does running Ceph fundamentally change your backup strategy compared to traditional storage solutions? Specifically, I'm interested in: - Are you backing up at the RBD level, or relying on application-level backups? - How do you handle cross-site replication for disaster recovery? - Any tools or practices you'd recommend for automated backup scheduling? Regards, [image] Anthony Fecarotta Founder & President [image] anthony(a)linehaul.ai <mailto:anthony@linehaul.ai> [image] 224-339-1182 [image] (855) 625-0300 [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 [image] www.linehaul.ai <http://www.linehaul.ai/> [image] <http://www.linehaul.ai/> [image] <https://www.linkedin.com/in/anthony-fec/>

5 6

Re: cluster without quorum
by Eugen Block 13 Aug '25

13 Aug '25

Hi, I think the ceph-users list is more appropriate for your request, I removed the dev list from the recipients If I understand correctly, you should still have one surviving monitor, is that correct? The quickest way to restore the quorum would probably be to edit the monmap (make a backup) by removing the dead monitors and startup the surviving monitor. This would give you a single-mon cluster, from there you could then redeploy the other monitors and restore OSD functionality. > I tried reinstalling the system on one node and recreating a monitor > using the data taken from /var/lib/ceph/osd/, but I don't see the > OSDs. This is the situation… Is this a cephadm managed cluster? In that case the OSDs have their data directory in /var/lib/ceph/{CLUSTER_FSID}/osd.{OSD_ID}/. If it's not cephadm managed, you'll need to provide more information with details and logs etc. Regards, Eugen Zitat von FRANCESCO NAPOLI <francesco.napoli(a)cnr.it>: > Good morning, Ceph was installed on Ubuntu Server 24.04 with > cephadm. The cluster consists of 4 nodes; the first one also acted > as a monitor in the initial state, but as all the others were added, > in addition to being OSD hosts, they also acted as monitors. These > machines had the major flaw of not having the operating system disk > in RAID, and after 20 days, 3 nodes failed. The OSD disks are > fine... I tried reinstalling the system on one node and recreating a > monitor using the data taken from /var/lib/ceph/osd/, but I don't > see the OSDs. This is the situation… > thanks for the support. > > -- > Napoli Francesco > Gruppo Reti e Sistemi Informativi > Istituto di Fisiologia Clinica - CNR > Tel. 050.315.2344

1 0

Problem deploying ceph 19.2.3 on Rocky linux 9
by wodel youchi 13 Aug '25

13 Aug '25

Hi, I am having issues trying to deploy ceph 19.2.3 on rocky linux 9. 1 - I cannot install ceph-common : it needs openssl 3.4, rocky 9 offers only openssl 3.2 2 - I bootstraped my cluster like this : cephadm bootstrap \ --mon-ip 10.200.15.10 \ --cluster-network 10.210.0.0/16 \ --initial-dashboard-user admin \ --initial-dashboard-password 'admin' \ --ssh-private-key /root/.ssh/id_rsa \ --ssh-public-key /root/.ssh/id_rsa.pub it did bootstrap, but when I try adding other hosts, I cannot, it has something to do with ssh. I did copy id.rsa.pub on the other nodes via ssh-copy-id, and I can ssh without password to them. but I cannot ssh to them via cephadm shell. and when i try to add a host I get : Aug 11 15:29:06 hyper01 ceph-mgr[35633]: [cephadm ERROR cephadm.ssh] Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError Aug 11 15:29:06 hyper01 ceph-mgr[35633]: log_channel(cephadm) log [ERR] : Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: 2025-08-11T14:29:06.509+0000 7f27b44ed640 -1 log_channel(cephadm) log [ERR] : Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Traceback (most recent call last): Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: await options.waiter Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: asyncio.exceptions.CancelledError Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: During handling of the above exception, another exception occurred: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Traceback (most recent call last): Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: return fut.result() Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: asyncio.exceptions.CancelledError Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: The above exception was the direct cause of the following exception: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: Traceback (most recent call last): Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: yield Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[self.mgr.tkey.name], Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: return await asyncio.wait_for( Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: raise exceptions.TimeoutError() from exc Aug 11 15:29:06 hyper01 ceph-42cab8b0-76bc-11f0-b9ed-7cc255beecde-mgr-hyper01-rzwgry[35624]: asyncio.exceptions.TimeoutError Aug 11 15:29:06 hyper01 ceph-mgr[35633]: [cephadm ERROR orchestrator._interface] Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1775, in _check_valid_addr out, err, code = self.wait_async(CephadmServe(self)._run_cephadm( File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/serve.py", line 1595, in _run_cephadm await self.mgr.ssh._remote_connection(host, addr) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _remote_connection raise File "/lib64/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 214, in redirect_log raise HostConnectionError( cephadm.ssh.HostConnectionError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3502, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1800, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1789, in _check_valid_addr raise OrchestratorError(str(e)) orchestrator._interface.OrchestratorError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Aug 11 15:29:06 hyper01 ceph-mgr[35633]: log_channel(cephadm) log [ERR] : Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Traceback (most recent call last): File "/lib/python3.9/site-packages/asyncssh/connection.py", line 440, in _connect await options.waiter asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib64/python3.9/asyncio/tasks.py", line 490, in wait_for return fut.result() asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 197, in redirect_log yield File "/usr/share/ceph/mgr/cephadm/ssh.py", line 174, in _remote_connection conn = await asyncssh.connect(addr, username=self.mgr.ssh_user, client_keys=[ self.mgr.tkey.name], File "/lib/python3.9/site-packages/asyncssh/connection.py", line 8170, in connect return await asyncio.wait_for( File "/lib64/python3.9/asyncio/tasks.py", line 492, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1775, in _check_valid_addr out, err, code = self.wait_async(CephadmServe(self)._run_cephadm( File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async return self.event_loop.get_result(coro, timeout) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result return future.result(timeout) File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result return self.__get_result() File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/serve.py", line 1595, in _run_cephadm await self.mgr.ssh._remote_connection(host, addr) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 182, in _remote_connection raise File "/lib64/python3.9/contextlib.py", line 137, in __exit__ self.gen.throw(typ, value, traceback) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 214, in redirect_log raise HostConnectionError( cephadm.ssh.HostConnectionError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3502, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1800, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1789, in _check_valid_addr raise OrchestratorError(str(e)) orchestrator._interface.OrchestratorError: Failed to connect to hyper02 (10.200.15.11): TimeoutError() Log: Opening SSH connection to 10.200.15.11, port 22 [conn=7] Connected to SSH server at 10.200.15.11, port 22 [conn=7] Local address: 10.200.15.10, port 53326 [conn=7] Peer address: 10.200.15.11, port 22 [conn=6, chan=10] Requesting new SSH session [conn=6, chan=10] Command: which python3 [conn=6, chan=10] Received exit status 0 [conn=6, chan=10] Received channel close [conn=6, chan=10] Channel closed [conn=6, chan=11] Requesting new SSH session [conn=6, chan=11] Command: /usr/bin/python3 /var/lib/ceph/42cab8b0-76bc-11f0-b9ed-7cc255beecde/cephadm.1a8853661a9c1798390b8e8d13c27688c1b1327a075745af2ee40ac466f0ac36 --timeout 895 gather-facts [conn=6, chan=11] Received exit status 0 [conn=6, chan=11] Received channel close [conn=6, chan=11] Channel closed [conn=6, chan=12] Requesting new SSH session [conn=6, chan=12] Command: ls /etc/sysctl.d [conn=6, chan=12] Received exit status 0 [conn=6, chan=12] Received channel close [conn=6, chan=12] Channel closed [conn=7] Aborting connection [conn=7] Connection closed Regards.

2 1

Performance scaling issue with multi-SSD (CrimsonOSD/Seastore)
by Ki-taek Lee 13 Aug '25

13 Aug '25

Hello Ceph community, I am evaluating Crimson OSD + Seastore performance for potential deployment in a distributed storage environment. With BlueStore, I have been able to achieve satisfying performance levels in my FIO tests for 4K random read/write IOPS. However, when testing Crimson OSD + Seastore, I observed that 4K random read/write IOPS do not scale as expected when increasing the number of SSDs/OSDs. The performance plateaus beyond a certain point or is much lower than expected. (See attached test results.) Test Environment: - Cluster: 8 clients, 1 OSD - Hardware: 40-core CPUs, 377 GiB DRAM - Image SHA (quay.io): e0543089a9e9cae97999761059eaccdf6bb22e9e - Configuration parameters: osd_memory_target = 34359738368 crimson_osd_scheduler_concurrency = 0 seastore_max_concurrent_transactions = 16 crimson_osd_obc_lru_size = 8192 seastore_cache_lru_size = 16G seastore_obj_data_write_amplification = 4 seastore_journal_batch_capacity = 1024 seastore_journal_batch_flush_size = 256M seastore_journal_iodepth_limit = 16 seastore_journal_batch_preferred_fullness = 0.8 seastore_segment_size = 128M seastore_device_size = 512G seastore_block_create = true seastore_default_object_metadata_reservation = 1073741824 rbd_cache = false rbd_cache_writethrough_until_flush = true rbd_op_threads = 16 Replication policy: - 4096 PGs, no replication (only 1 copy) Test Results: 1 SSD test (varying number of allocated CPUs, alien threads = 26-29, 36-39): num CPU | 4k randread | 4k randwrite | Allocated CPU sets 2 | 126772 | 14830 | 0-1 4 | 107860 | 16451 | 0-3 6 | 113741 | 17019 | 0-5 8 | 132060 | 16099 | 0-7 SSD scaling test (2 CPUs per SSD): OSD CPU mapping: OSD.0 (0-1), OSD.1 (10-11), OSD.2 (2-3), OSD.3 (12-13), ..., OSD.15 (34-35), Alien threads (26-29, 36-39) num SSD | 4k randread | 4k randwrite 4 | 861273 | 22360 8 | 1022793 | 22786 12 | 1019161 | 21211 16 | 927570 | 20502 SSD scaling test (1 CPU per SSD): OSD CPU mapping: OSD.0 (0), OSD.1 (10), OSD.2 (2), OSD.3 (12), ..., OSD.15 (24), Alien CPUs: 1, 11, 3, 13, ..., 15, 25 num SSD | 4k randread | 4k randwrite 4 | 936685 | 13730 8 | 1048204 | 18259 12 | 922727 | 23078 16 | 987838 | 30792 Questions: 1. Since Seastore is still under active development, are there any known unresolved performance issues that could explain this scaling behavior? 2. Are there recommended tuning parameters for improving small-block read scalability in multi-SSD configurations? 3. Regarding alien threads, are there best practices for CPU pinning or NUMA-aware placement that have shown measurable improvements? 4. Any additional guidance for maximizing IOPS with Crimson OSD + Seastore would be greatly appreciated. My goal is to be ready to switch from BlueStore to Crimson + Seastore after it becomes stable and shows reasonable performance compared to BlueStore, so I’d like to understand the current limitations and tuning opportunities. Thank you, Ki-taek Lee

4 3

Windows support for Ceph
by Anthony Middleton 12 Aug '25

12 Aug '25

The Ceph Steering Committee (CSC) is evaluating Windows support for Ceph. Please answer the question(s) in the survey below to help guide the CSC's decision. Survey link - https://forms.gle/b6WBry3i6JuLrVfP9 *The deadline to complete this survey is August 25th, 2025. We appreciate your participation.* If you have any questions, please contact Anthony Middleton at community-manager(a)ceph.io.

1 0

Ceph subreddit banned?
by Philipp Hocke 12 Aug '25

12 Aug '25

Hi everyone, I just noticed that the Ceph subreddit has been banned: "This community was banned for violating Rule 2. You agree that use of this site constitutes acceptance of Reddit’s User Agreement and acknowledge our Privacy Policy." Does anyone know what's going on? Best regards Philipp -- Philipp Hocke Leiter Linux Systemadministration / Head Of TechOps CM4ALL GmbH Im Mediapark 6A - 50670 Köln / Cologne Phone +49-(0)221-6601-0 <https://outlook.office.com/mail/sentitems> Fax +49-(0)221-6601-1011 <https://outlook.office.com/mail/sentitems> E-Mail: philipp.hocke(a)cm4all.com Internet: www.cm4all.com

2 1

Announcing go-ceph v0.32.0
by Sven Anderson 12 Aug '25

12 Aug '25

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.32.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1

Debugging OSD cache thrashing
by Hector Martin 11 Aug '25

11 Aug '25

Hi all, I have a small 3-node cluster (4 HDD + 1 SSD OSD per node, ceph version 19.2.2) and noticed that during snaptrim ops (?), the OSDs max out CPU usage and cluster performance plummets. Most of the CPU usage was accounted as "system", while actual disk I/O usage was low, so that didn't sound right. I perf traced the system, and found that most of the usage is in __arch_copy_to_user in the kernel, in read() syscalls. That sounds like the OSD is thrashing buffered block device reads which the kernel satisfies from the page cache (hence no real I/O load, most of the CPU usage is from data copies), so it's repeatedly reading the same disk blocks, which doesn't sound right. I increased osd_memory_target to 2.4G (from 1.2G) live with `ceph config`, and CPU usage immediately dropped to near zero. However, waiting a bit eventually the CPU thrashing returns after memory usage increases. Restarting an OSD has a similar effect. I believe that something is wrong with the OSD bluestore cache allocation/flush policy, and when the cache becomes full it starts thrashing reads instead of evicting colder cached data (or perhaps some cache bucket is starving another cache bucket of space). I would appreciate some hints on how to debug this. Are there any cache stats I should be looking at, or info on how the cache is partitioned? Here is a perf call trace of the thrashing: > - OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) > - 95.04% ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&) > - OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&) > - 95.03% PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&) > - PGBackend::handle_message(boost::intrusive_ptr<OpRequest>) > - ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>) > - 88.86% ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr<OpRequest>, ECSubWrite&, ZTracer::Trace const&, ECListener&) > - 88.78% non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<OpRequest>) > - BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusiv > - 88.75% BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*) > - 88.71% BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&) > - 88.69% BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, std > - 88.68% BlueStore::_wctx_finish(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, BlueStore::Writ > - 88.65% BlueStore::Collection::load_shared_blob(boost::intrusive_ptr<BlueStore::SharedBlob>) > - 88.64% RocksDBStore::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::ch > - 88.61% rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*) > - rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions&) > - 88.43% rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::PinnableWideColumns*, std > - 88.41% rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Sl > - rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, bo > - 88.32% rocksdb::BlockBasedTable::FullFilterKeyMayMatch(rocksdb::FilterBlockReader*, rocksdb::Slice const&, bool, rocksdb::SliceTransfor > - rocksdb::FullFilterBlockReader::MayMatch(rocksdb::Slice const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocks > - 88.29% rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::GetOrReadFilterBlock(bool, rocksdb::GetContext*, rocksdb > - rocksdb::FilterBlockReaderCommon<rocksdb::ParsedFullFilterBlock>::ReadFilterBlock(rocksdb::BlockBasedTable const*, rocksdb::Fi > - rocksdb::Status rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePrefetchBuffer*, rocks > - rocksdb::Status rocksdb::BlockBasedTable::MaybeReadBlockAndLoadToCache<rocksdb::ParsedFullFilterBlock>(rocksdb::FilePref > - 88.25% rocksdb::BlockFetcher::ReadBlockContents() [ note: split, the rest is + 19.42% rocksdb::VerifyBlockChecksum] > - 68.80% rocksdb::RandomAccessFileReader::Read(rocksdb::IOOptions const&, unsigned long, unsigned long, rocksdb::Sli [ note: at this level the call trace splits into 4, but it all leads to the same place ] > - 35.99% 0xaaaac0992e40 > BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const > BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*) > - KernelDevice::read_random(unsigned long, unsigned long, char*, bool) > - 35.98% __libc_pread > - el0_svc > - invoke_syscall > - 35.97% __arm64_sys_pread64 > - 35.96% vfs_read > - blkdev_read_iter > - 35.93% filemap_read > - 35.28% copy_page_to_iter > 35.01% __arch_copy_to_user - Hector

7 17

How to change to RocksDB LZ4 after upgrade to Ceph 19
by Niklas Hambüchen 11 Aug '25

11 Aug '25

Release notes https://ceph.io/en/news/blog/2024/v19-2-0-squid-released/#highlights say: > BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. On one EC pool with many files I have such warnings: osd.6 spilled over 8.0 GiB metadata from 'db' device (36 GiB used of 60 GiB) to slow device and hope that the LZ4 compression might help with that. What do I need to do to "migrate" my OSDs to that? I already upgrade the cluster to Ceph 19. Running grep 'Options.compression: ' /var/log/ceph/ceph-osd.6.log already shows the switch from rocksdb: Options.compression: NoCompression to rocksdb: Options.compression: LZ4 in the logs. But does restarting the OSDs already automtaically compress everything with LZ4, or do I need to run some operation such as ceph daemon osd.6 compact or similar to make sure that existing data is actually compressed? Also, could anybody clarify whether this setting is `compression_algorithm` from https://docs.ceph.com/en/squid/rados/operations/pools/#setting-pool-values https://docs.ceph.com/en/squid/rados/configuration/bluestore-config-ref/#co… or if that's something different (e.g. if that's "actual data" instead of "metadata")? I suspect it's different, because `ceph config show-with-defaults osd.6 | grep compression` reveals: bluestore_compression_algorithm snappy bluestore_compression_mode none bluestore_rocksdb_options compression=kLZ4Compression,... From this it looks like `bluestore_compression*` is the "Inline compression" for data (https://docs.ceph.com/en/reef/rados/configuration/bluestore-config-ref/#inl…) and `bluestore_rocksdb_options` is what the "RocksDB compression" is about. Still the question remains on how to bring all existing data over to be compressed. Thanks!

3 2