- ceph-users - lists.ceph.io

capabilities needed for subvolume management?
by Robert Sander 12 Sep '25

12 Sep '25

Hi, what capabilities does a cephx key need to manage subvolumes in a filesystem? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

v20.1.0 Tentacle RC0 released
by Yuri Weinstein 12 Sep '25

12 Sep '25

This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0 Feature highlights: * RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4

5 10

Re: snap-schedule not running
by Sophonet 12 Sep '25

12 Sep '25

Thanks Eugen, after enabling debugging temporarily, I have seen an error in the logs when creating the snapshot with mkdir snapshot in .snap including a suspicious line mds.0.server is a descendent of a subvolume dir Indeed, when doing a snapshot on the root folder of the filesystem (which works), the log files tell me the inode of the root directory and ceph daemon mds.MDSID dump inode 0x1 (for the root folder) outputs many information in json, including „is_subvolume“: true Indeed, for a different filesystem, where snapshots work as expected in root folder and subfolders, the same corresponding information from the root inode says „is_subvolume“: false. How can the root folder in a filesystem be a subvolume and what can I do to fix this? Also I am puzzled what has caused this problem… Thanks, Sophonet

2 1

[18.2.4 Reef] Using RAID1 as Metadata Device in Ceph – Risks and Recommendations
by Alex from North 11 Sep '25

11 Sep '25

Hi all! I’ve modified Ceph to allow using a RAID1 array as the metadata device. Specifically, I've updated ceph_volume/util/device.py and ceph_volume/util/disk.py to recognize the "raid1" device type as valid device for OSDs and metadata storage. The changes are listed below. My questions: - What are the risks or issues which may arise from using RAID1 for metadata? (e.g., performance, reliability, data integrity) - Why this could be a bad idea, or in what scenarios could it be beneficial? My test setup: - Ceph version: 18.2.4 Reef (stable) - OS: Ubuntu 22.04.5 LTS - Hardware: 1 host, RAID1 array on two NVMe SSDs (/dev/nvme0n1 and /dev/nvme0n2, 19.98 GiB) via mdadm for block.db, 4 HDDs for OSD data - Additional details: Tested on a small cluster on VMs; no noticeable performance changes so far, but I’m concerned about long-term implications. Code changes: diff --git a/src/ceph-volume/ceph_volume/util/device.py b/src/ceph-volume/ceph_volume/util/device.py index 1b52774d1a1..148a8e326f8 100644 --- a/src/ceph-volume/ceph_volume/util/device.py +++ b/src/ceph-volume/ceph_volume/util/device.py @@ -237,7 +237,7 @@ class Device(object): self.disk_api = dev device_type = dev.get('TYPE', '') # always check is this is an lvm member - valid_types = ['part', 'disk', 'mpath'] + valid_types = ['part', 'disk', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') if device_type in valid_types: @@ -489,7 +489,7 @@ class Device(object): elif self.blkid_api: api = self.blkid_api if api: - valid_types = ['disk', 'device', 'mpath'] + valid_types = ['disk', 'device', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') return self.device_type in valid_types diff --git a/src/ceph-volume/ceph_volume/util/disk.py b/src/ceph-volume/ceph_volume/util/disk.py index 2984c391d06..5dafd6dc62a 100644 --- a/src/ceph-volume/ceph_volume/util/disk.py +++ b/src/ceph-volume/ceph_volume/util/disk.py @@ -362,7 +362,7 @@ def is_device(dev): TYPE = lsblk(dev).get('TYPE') if TYPE: - return TYPE in ['disk', 'mpath'] + return TYPE in ['disk', 'mpath', 'raid1'] # fallback to stat return _stat_is_device(os.lstat(dev).st_mode) and not is_partition(dev) Any feedback is really appreciated as all of us care about not to lost our precious bytes :)

3 6

Mirroring images per snapshot to a different pool on remote site
by Emmanuel Lacour 11 Sep '25

11 Sep '25

Dear ceph users, I would like to enable mirroring of a site A to a site B, using per images mirroring and snapshots, but I would like to point to a different pool name on site B (there is already a pool with same name used on site B and I want to target another one with less replication). It seems this is not supported (by reading the doc), but I ask here in case of ...

1 0

OSD socket closed
by Samuel Moya Tinoco 10 Sep '25

10 Sep '25

Hi everyone, Recently i had a problem with my pve cluster and while looking through logs we find this kind of logs in every node Aug 29 10:44:26 pve01-poz kernel: [697358.187283] libceph: osd17 (1)10.0.0.3:6810 socket closed (con state OPEN) Aug 29 10:45:43 pve01-poz kernel: [697434.733343] libceph: osd1 (1)10.0.0.1:6837 socket closed (con state OPEN) Aug 29 10:46:54 pve01-poz kernel: [697506.414196] libceph: osd15 (1)10.0.0.1:6836 socket closed (con state OPEN) Aug 29 10:47:56 pve01-poz kernel: [697567.855454] libceph: osd4 (1)10.0.0.1:6802 socket closed (con state OPEN) Aug 29 10:49:08 pve01-poz kernel: [697639.497740] libceph: osd4 (1)10.0.0.1:6802 socket closed (con state OPEN) Aug 29 10:51:50 pve01-poz kernel: [697802.060504] libceph: osd12 (1)10.0.0.3:6811 socket closed (con state OPEN) Aug 29 10:55:37 pve01-poz kernel: [698028.664214] libceph: osd1 (1)10.0.0.1:6837 socket closed (con state OPEN) The problem we had was with corosync that is using another network and other ethernet ports. But we'd like to know if this logs are normal and it's just part of how Ceph work or if we should dig deep to check what is causing this logs Thank you in advance for your time and help Samuel Moya Tinoco Departamento de Sistemas y Redes Móvil: (+34) 606985997 smoya(a)vivelibre.es<mailto:smoya@vivelibre.es> ViveLibre C/ La Orotava 4 28660 Boadilla del Monte Madrid www.vivelibre.es<http://www.vivelibre.es/> [cid:image001.png@01DC219E.CAC50090] Soluciones inteligentes para la autonomía personal "Tanto este mensaje como todos los posibles documentos adjuntos al mismo son confidenciales y están dirigidos exclusivamente a los destinatarios de los mismos. Por favor, si Usted no es uno de dichos destinatarios, notifíquenos este hecho y elimine el mensaje de su sistema. Queda prohibida la copia, difusión o revelación de su contenido a terceros sin el previo consentimiento por escrito de VIVELIBRE AUTONOMÍA PERSONAL S.L.U. (VIVELIBRE). En caso contrario, vulnerará la legislación vigente. De conformidad con lo establecido en el Reglamento (UE) 2016/679, General de Protección de Datos, le informamos de que sus datos son objeto de tratamiento por VIVELIBRE, en calidad de Responsable del Tratamiento. VIVELIBRE tratará sus datos con la finalidad de mantener la relación contractual, gestionar su solicitud, así como remitirle comunicaciones de carácter comercial relacionadas con su ámbito de actividad y los servicios prestados. Si desea ejercitar sus derechos de acceso, rectificación, supresión, limitación, oposición o portabilidad, puede dirigirse a la dirección postal Calle de la Orotava 4, 28660, Boadilla del Monte, (Madrid) o a la dirección de correo electrónico protecciondedatos(a)vivelibre.es. Para obtener más información sobre cómo tratamos sus datos, consulta nuestra Política de Privacidad en Política de Privacidad - Vivelibre."

4 3

test
by wissem mimouna 10 Sep '25

10 Sep '25

test

1 0

peer... is using msgr V1 protocol ?
by lejeczek 09 Sep '25

09 Sep '25

Hi guys. I'm getting following in logs: ... 2025-09-09T11:49:00.866+0000 7f3976084640 -1 --2- 10.1.1.61:0/3730235752 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3d59c400 0x55ee3ec6a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol --2- 10.1.1.61:0/3730235752 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3d59c400 0x55ee3ec6a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol 2025-09-09T11:49:00.876+0000 7f3976885640 -1 --2- 10.1.1.61:0/1308618359 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3e85a000 0x55ee3e860000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol --2- 10.1.1.61:0/1308618359 >> [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] conn(0x55ee3e85a000 0x55ee3e860000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:10.1.1.62:6812/3478340378,v1:10.1.1.62:6813/3478340378] is using msgr V1 protocol ... This three-node containerized cluster. Only one peer-node logs these error - its IP is 10.1.1.61 - xx.62 & xx.63 are other two peer-nodes. Where/how would you start troubleshooting this? many thanks, L.

2 2

ceph restful key permissions
by BASSAGET Cédric 09 Sep '25

09 Sep '25

Hello, I'm trying to create a restful api key with read-only permissions, but I didn't find the way to do that. Is that doable ? Regards Cédric

2 1

Ceph OSD with migrated db device fails after upgrade to 19.2.3 from 18.2.7
by Soenke Schippmann 09 Sep '25

09 Sep '25

Hi, after upgrading one of our ceph clusters from 18.2.7 to 19.2.3 some OSDs fail to start. For these OSDs, db devices were moved manually months ago from a partition to a lvm volume. OSD log shows: 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 set uid:gid to 167:167 (ceph:ceph) 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable), process ceph-osd, pid 7 2025-09-04T11:38:22.055+0000 7fec1bbc4740 0 pidfile_write: ignore empty --pid-file 2025-09-04T11:38:22.055+0000 7fec1bbc4740 1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open path /var/lib/ceph/osd/ceph-256/block 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 bdev(0x556f24a77400 /var/lib/ceph/osd/ceph-256/block) open stat got: (1) Operation not permitted 2025-09-04T11:38:22.055+0000 7fec1bbc4740 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-256: (2) No such file or directory Links to block and block.db within the osd path get deleted after each startup attempt. Recreating the links manually does not help. ceph-bluestore-tool fsck --path... shows no errors if links to block and block.db are recreated. Running "ceph-volume activate --osd-id 256" manually within cephadm shell fails with the follwing error: --> Failed to activate via LVM: could not find db with uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f ceph-volume lvm list shows outdated db uuid and db device: ===== osd.256 ====== [block] /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block device /dev/ceph-402b182b-c5bd-416d-bde6-668e772b0c4c/osd-block-93811afc-17a3-4458-8e00-506eb9c92cb0 block uuid GRd7Zo-dXdx-23Wf-507d-dHPw-6UOA-oaf7Qy cephx lockbox secret cluster fsid 1f3b3198-08b1-418c-a279-7050a2eb1ce3 cluster name ceph crush device class None db device /dev/sdai1 db uuid 6d676bcd-1f3c-e740-8fdf-6a5156605a3f encrypted 0 osd fsid 93811afc-17a3-4458-8e00-506eb9c92cb0 osd id 256 type block vdo 0 devices /dev/sdc [db] /dev/sdai1 PARTUUID 6d676bcd-1f3c-e740-8fdf-6a5156605a3f db was migrated from partition /dev/sdai1 to lvm volume ceph-blockdb-01/osd-db-01 on /dev/sdaa months ago and running fine with Ceph 18.2. Migration was done manually by using "ceph-bluestore-tool bluefs-bdev-migrate" (ceph-volume lvm migrate failed though). Is there any way to fix this? Best, Sönke -- Sönke Schippmann Universität Bremen Dezernat 8 - IT Service Center Referat 82 Serverbetrieb Büroanschrift: Universität Bremen Dez. 8-Bi, SFG 1390 Enrique-Schmidt-Str. 7 28359 Bremen E-Mail: schippmann(a)uni-bremen.de Tel: +49 421 218-61327 Fax: +49 421 218-98-61327 http://www.uni-bremen.de/zfn/

5 6