Is it possible to use Ceph as a root filesystem for a pxe booted host? Thanks
Hello, Jeremy Hansen wrote: : Is it possible to use Ceph as a root filesystem for a pxe booted host? I am not sure about CephFS, but as for Ceph RBD, it should definitely be possible to get an IP address and run "rbd map" from the initrd, and then use /dev/rbd0 as a root device. -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall
A while ago I had done this with iSCSI, using iBFT, the advantage being the NIC itself supports such boot, the disadvantage is you need to use iSCSI. I recall PXE can bring an image over TFTP, so assuming you want to load that image into an RBD device for persistence? Looks like this was discussed here: https://www.spinics.net/lists/ceph-users/msg66370.html -- Alex Gorbachev iss-integration.com On Wed, Jan 3, 2024 at 10:10 PM Jeremy Hansen <jeremy@skidrow.la> wrote:
Is it possible to use Ceph as a root filesystem for a pxe booted host?
Thanks
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We also had a look at this a few years ago when we flashed almost all servers to boot iPXE from the nic directly. The idea was as follows: iPXE -> http -> get kernel + initramfs -> mount rootfs Our idea was to use RBD as a disk for the server, but to the last of my knowledge, there is no support in the linux kernel for something like linux root=ceph://mon1,mon2,mon3,key=keytouse/rbd/pool-x/image-y Which would be needed for a ceph based rootfs. It would be amazing to have that though, a decentralised storage system that is able to be booted from. Above description would actually be more suitable to the requested cephfs booting, as NFS root also often splits the kernel+initramfs. The RBD case would potentially require boot loader support, if we assume that the RBD image contains something like grub - so a tiny bit more complicated. BR and wishing you all the best for this approach, Nico Alex Gorbachev <ag@iss-integration.com> writes:
A while ago I had done this with iSCSI, using iBFT, the advantage being the NIC itself supports such boot, the disadvantage is you need to use iSCSI.
I recall PXE can bring an image over TFTP, so assuming you want to load that image into an RBD device for persistence? Looks like this was discussed here: https://www.spinics.net/lists/ceph-users/msg66370.html
-- Sustainable and modern Infrastructures by ungleich.ch
My €0.02 for what it's worth(less). I've been doing RBD-based VMs under libvirt with no problem. In that particular case, the ceph RBD base images are being overlaid cloud- style with a an instance-specific qcow2 image and the RBD is just part of my storage pools. For a physical machine, I'd probably be content to use the Ceph NFS share for the OS and PXE boot itself. I could always mount a ceph filesystem once the OS was up if I wanted to. If I'm not mistaken, getting ceph to boot directly would require a custom ramdisk with ceph driver installed, whereas an NFS-supporting ramdisk is pretty much off-the-shelf. Since ceph-ganesha NFS seems pretty reliable and performant, I'd be happy with that. Tim On Thu, 2024-01-04 at 17:16 +0100, Nico Schottelius wrote:
We also had a look at this a few years ago when we flashed almost all servers to boot iPXE from the nic directly. The idea was as follows:
iPXE -> http -> get kernel + initramfs -> mount rootfs
Our idea was to use RBD as a disk for the server, but to the last of my knowledge, there is no support in the linux kernel for something like
linux root=ceph://mon1,mon2,mon3,key=keytouse/rbd/pool-x/image-y
Which would be needed for a ceph based rootfs.
It would be amazing to have that though, a decentralised storage system that is able to be booted from. Above description would actually be more suitable to the requested cephfs booting, as NFS root also often splits the kernel+initramfs.
The RBD case would potentially require boot loader support, if we assume that the RBD image contains something like grub - so a tiny bit more complicated.
BR and wishing you all the best for this approach,
Nico
Alex Gorbachev <ag@iss-integration.com> writes:
A while ago I had done this with iSCSI, using iBFT, the advantage being the NIC itself supports such boot, the disadvantage is you need to use iSCSI.
I recall PXE can bring an image over TFTP, so assuming you want to load that image into an RBD device for persistence? Looks like this was discussed here: https://www.spinics.net/lists/ceph-users/msg66370.html
-- Sustainable and modern Infrastructures by ungleich.ch _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Alex Gorbachev
-
Jan Kasprzak
-
Jeremy Hansen
-
Nico Schottelius
-
Tim Holloway