- ceph-users - lists.ceph.io

Re: resize underlying volume - disk size increase ?
by lejeczek 15 Jan '26

15 Jan '26

is this... -> $ systemctl stop ceph-9f4f9dba-72c7-11f0-8052-525400519d29(a)osd.10.service -> $ ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.10/ inferring bluefs devices from bluestore path 1 : device size 0x4affc00000(300 GiB) : using 0x36f49bd000(220 GiB) -> $ ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.10/ inferring bluefs devices from bluestore path 1 : device size 0x4affc00000(300 GiB) : using 0x36f49bd000(220 GiB) Expanding DB/WAL... 1 : nothing to do, skipped not weird(wrong) - would not all agree? -> $ ceph osd df ... 10 hdd 0.29300 1.00000 300 GiB 219 GiB 218 GiB 567 KiB 632 MiB 81 GiB 72.91 0.99 165 up -> $ ceph-volume lvm list ... ====== osd.10 ====== ... devices /dev/vdc ... -> $ fdisk -l /dev/vdc Disk /dev/vdc: 400 GiB, 429496729600 bytes, 104857600 sectors Something, somewhere Ceph is ignoring, not picking up...? From what I can find 'ceph-volume' does not offer re-sizing volumes.

1 0

Re: Dual stack support in squid?
by Robert Sander 15 Jan '26

15 Jan '26

Hi, Am 15.01.26 um 3:24 PM schrieb Burkhard Linke via ceph-users: > Nonetheless, since most clients are already running with dual stack (the > k8s cluster will be the last and probably most "interesting" setup..), > let's adopt the migration and target ipv6 only: > > - set ms_bind_ipv4 to false and ms_bind_ipv6 to true > > - delete and re-add mons one after another > > - process the other components like OSDs etc. As a cluster cannot run dual stack there is unfortunately no way to migrate from IPv4 to IPv6 online. You may be able to edit the MON DBs on all MONs manually when the cluster is completely offline and replace the MON IPs there. But I have no experience in doing this. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: resize underlying volume - disk size increase ?
by Eugen Block 15 Jan '26

15 Jan '26

First: yes, the OSD has to be down to be able to operate on it. Second: for cephadm managed daemons you'll need to use the cephadm shell, so something like (omit the FSID because the full path is mapped into the container): cephadm shell -- ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/osd.9/ Third: there's a bluefs-bdev-expand command for ceph-bluestore-tool. That could help here (IIUC, your main device has more space, no migration of DB/WAL). I haven't done this in a while (never on a production cluster, concurring with Janne's response), but I'm pretty sure it worked. So stop osd (set noout), resize PV (which you already have), run bluestore-tool (within cephadm shell) to expand the device, then start the osd (and unset noout). Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > In my lab ceph is deployed with cephadm - so, containerized env, when I do: > > -> $ ceph-bluestore-tool bluefs-bdev-sizes --path > /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.9/ > ... > I get a coredump - on each node, each osd - as I thumbed through the > net I read somewhere that an 'osd' must be stopped for > 'ceph-bluestore-tool' to operate on it? > But how would that make sense ? not in my mind, but hey... or I'm > doing something forbidden there. > > I don't think a "migration" - loosely defined - applies to my > scenario as there is nothing to migration onto - disks which Cephs > uses "got" more space - no new/more disks/devices. > > Ceph sees those changes already: > -> $ ceph orch device ls > podster1.mine.priv /dev/vdc hdd 400G No 13m ago > Has a FileSystem, LVM detected > ... > podster3.mine.priv /dev/vdc hdd 300G No 13m ago > Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM > detected > > first above, is easy to guess - got extra 100GB. I did fiddled with > it actually "outside" of ceph, did 'pvresize' (PV which Ceph itself > crerated and used LVM) which did not cause any troubles, apparently. > So I was itching to do the 'lvresize' too but though Ceph's way > should be (existent) preferred - I can imagine even in _production_ > people do use.. > partitions/LVM - as opposed to whole disks/drives.. no? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: resize underlying volume - disk size increase ?
by Janne Johansson 15 Jan '26

15 Jan '26

> So I was itching to do the 'lvresize' too but though Ceph's > way should be (existent) preferred - I can imagine even in > _production_ people do use.. > partitions/LVM - as opposed to whole disks/drives.. no? Well, in my production setups, disks just never "grow", they get replaced, so having ceph work redo a disk with a bad setting by adding another and remaking the OSD is kind of how we do it. I get that it can be interesting to see how to live-grow what basically amounts to a database on a partition, but since we left filestore which was easy to expand in this manner for bluestore, this is a lot harder now, but it was never a real use-case for serious ceph clusters at our place to mess around with partition sizes in this way. Yes, we use whole disks, but ceph-volume makes them into lvm (with cryptsetup on top if asked) anyhow for other reasons than resizing support. -- May the most significant bit of your life be positive.

1 0

Re: resize underlying volume - disk size increase ?
by lejeczek 15 Jan '26

15 Jan '26

In my lab ceph is deployed with cephadm - so, containerized env, when I do: -> $ ceph-bluestore-tool bluefs-bdev-sizes --path /var/lib/ceph/9f4f9dba-72c7-11f0-8052-525400519d29/osd.9/ ... I get a coredump - on each node, each osd - as I thumbed through the net I read somewhere that an 'osd' must be stopped for 'ceph-bluestore-tool' to operate on it? But how would that make sense ? not in my mind, but hey... or I'm doing something forbidden there. I don't think a "migration" - loosely defined - applies to my scenario as there is nothing to migration onto - disks which Cephs uses "got" more space - no new/more disks/devices. Ceph sees those changes already: -> $ ceph orch device ls podster1.mine.priv /dev/vdc hdd 400G No 13m ago Has a FileSystem, LVM detected ... podster3.mine.priv /dev/vdc hdd 300G No 13m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected first above, is easy to guess - got extra 100GB. I did fiddled with it actually "outside" of ceph, did 'pvresize' (PV which Ceph itself crerated and used LVM) which did not cause any troubles, apparently. So I was itching to do the 'lvresize' too but though Ceph's way should be (existent) preferred - I can imagine even in _production_ people do use.. partitions/LVM - as opposed to whole disks/drives.. no?

1 0

Re: resize underlying volume - disk size increase ?
by Joachim Kraftmayer 15 Jan '26

15 Jan '26

hope it helps: https://docs.clyso.com/blog/ceph-osd-migrate-db-to-larger-ssd-flash-device/ joachim.kraftmayer(a)clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Do., 15. Jan. 2026 um 15:26 Uhr schrieb Burkhard Linke via ceph-users < ceph-users(a)ceph.io>: > Hi, > > On 1/15/26 15:12, lejeczek via ceph-users wrote: > > Hi guys. > > > > I do not suppose this happens in often - it's a lab in my case - when > > disks sizes, which Ceph uses increase. > > Is there a procedure - or howto/method if such thing is not > > approved/recommended - to have Ceph make use of such added disk space? > > > You can use the ceph-bluestore-tool to resize OSD _DB_ partitions, not > sure whether it also supports resizing the actual data partition. > > If in doubt the probably better method is draining the OSD, remove it > and recreate it afterwards. > > > Best regards, > > Burkhard > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: resize underlying volume - disk size increase ?
by Burkhard Linke 15 Jan '26

15 Jan '26

Hi, On 1/15/26 15:12, lejeczek via ceph-users wrote: > Hi guys. > > I do not suppose this happens in often - it's a lab in my case - when > disks sizes, which Ceph uses increase. > Is there a procedure - or howto/method if such thing is not > approved/recommended - to have Ceph make use of such added disk space? You can use the ceph-bluestore-tool to resize OSD _DB_ partitions, not sure whether it also supports resizing the actual data partition. If in doubt the probably better method is draining the OSD, remove it and recreate it afterwards. Best regards, Burkhard

1 0

Re: Dual stack support in squid?
by Burkhard Linke 15 Jan '26

15 Jan '26

Hi Adam, On 1/15/26 14:21, Adam Prycki via ceph-users wrote: > Hello, > > ceph doesn't support dual-stack networking. > Cluster components bind to either v4 or v6 address. > > It's the same with multiple subnets. You can have multiple subnets > configured in your cluster, but each component binds only to one of > these subnets. > > > - How does the mon process decide which address to use? > Lookup ms_bind_ipv4 and ms_bind_ipv6. (these options are exclusive, as > I mentioned above) > > Thanks for the information. https://docs.ceph.com/en/squid/rados/configuration/network-config-ref/ does not mention that the ms_bind_ipv4 an ms_bind_ipv6 options are exclusive, so I'm a little bit confused. I would have expected software like ceph to handle dual stack setups in 2026. If I remember correctly this was one of the goals of the MsgrV2 protocol. Nonetheless, since most clients are already running with dual stack (the k8s cluster will be the last and probably most "interesting" setup..), let's adopt the migration and target ipv6 only: - set ms_bind_ipv4 to false and ms_bind_ipv6 to true - delete and re-add mons one after another - process the other components like OSDs etc. Do Ceph components use their public address to bind _outgoing_ connections to a certain interface/IP address? If this is the case I do not see any sane and simple way to migrate to IPv6. Example: mon A: 1.2.3.1/24 (ceph public) and 1234::1/64 mon B: 1.2.3.2/24 and 1234::2/64 (ceph public) If mon A want to send packets to mon B, it can use its ipv6 address as source, which is _not_ the public address of the mon; vice versa for connections from mon B to mon A. in this setup a migration to IPv6 is straightforward, if all clients are capable of IPv6. Best regards, Burkhard

1 0

resize underlying volume - disk size increase ?
by lejeczek 15 Jan '26

15 Jan '26

Hi guys. I do not suppose this happens in often - it's a lab in my case - when disks sizes, which Ceph uses increase. Is there a procedure - or howto/method if such thing is not approved/recommended - to have Ceph make use of such added disk space? many thanks, L.

1 0

Re: Dual stack support in squid?
by Adam Prycki 15 Jan '26

15 Jan '26

Hello, ceph doesn't support dual-stack networking. Cluster components bind to either v4 or v6 address. It's the same with multiple subnets. You can have multiple subnets configured in your cluster, but each component binds only to one of these subnets. > - How does the mon process decide which address to use? Lookup ms_bind_ipv4 and ms_bind_ipv6. (these options are exclusive, as I mentioned above) Best regerds Adam Prycki On 15/01/2026 14:10, Burkhard Linke via ceph-users wrote: > Hi, > > > we are currently migrating our ipv4 infrastructure to ipv6, and maybe > later to ipv6 only. This also include our ceph clusters. > > My naive approach this is: > > 1. add a forth (ipv4) mon as safe guard and remove an existing mon > > 2. set ms_bind_ipv6 to true and add ipv6 networks to public_network > (both in ceph.conf and via ceph config) > > 3. manually re-add the removed mon with current map and key > > 4. repeat with other mons and afterwards restart osds and other services > to bind to ipv6, too > > > The re-added mon in step 3 does not bind to ipv6, only to ipv4. (both > port 3300 and 6789). According to the `ceph tell mon.XYZ config show` > output both the ms_bind_ipv6 and the public_network settings are > recognized by the process. > > > So the questions are: > > - Is dual stack supported for mons? The documentation does not mention > it explicitly... > > - Can a mon bind to multiple addresses at all, e.g. one IPv4 and one IPv6? > > - How does the mon process decide which address to use? > > > Best regards, > > Burkhard > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0