libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
I am seeing these messages when booting from RBD and booting hangs there. libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping However, Ceph Health is OK, so I have no idea what is going on. I reboot my 3 node cluster and it works again for about two weeks. How can I find out more about this issue, how can I dig deeper? Also there has been at least one report about this issue before on this mailing list - "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" - but no solution has been presented. This report was from 2018, so no idea if this is still an issue for Dyweni the original reporter. If you read this, I would be happy to hear how you solved the problem. Cheers Markus
On Sun, Apr 25, 2021 at 12:37 AM Markus Kienast <mark@trickkiste.at> wrote:
I am seeing these messages when booting from RBD and booting hangs there.
libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
However, Ceph Health is OK, so I have no idea what is going on. I reboot my 3 node cluster and it works again for about two weeks.
How can I find out more about this issue, how can I dig deeper? Also there has been at least one report about this issue before on this mailing list - "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" - but no solution has been presented.
This report was from 2018, so no idea if this is still an issue for Dyweni the original reporter. If you read this, I would be happy to hear how you solved the problem.
Hi Markus, What versions of ceph and the kernel are in use? Are you also seeing I/O errors and "missing primary copy of ..., will try copies on ..." messages in the OSD logs (in this case osd2)? Thanks, Ilya
On Sun, Apr 25, 2021 at 11:42 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Sun, Apr 25, 2021 at 12:37 AM Markus Kienast <mark@trickkiste.at> wrote:
I am seeing these messages when booting from RBD and booting hangs there.
libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
However, Ceph Health is OK, so I have no idea what is going on. I reboot my 3 node cluster and it works again for about two weeks.
How can I find out more about this issue, how can I dig deeper? Also there has been at least one report about this issue before on this mailing list - "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" - but no solution has been presented.
This report was from 2018, so no idea if this is still an issue for Dyweni the original reporter. If you read this, I would be happy to hear how you solved the problem.
Hi Markus,
What versions of ceph and the kernel are in use?
Are you also seeing I/O errors and "missing primary copy of ..., will try copies on ..." messages in the OSD logs (in this case osd2)?
For the sake of archives, the "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" instance has been fixed in 12.2.12, 13.2.5 and 14.2.0: https://tracker.ceph.com/issues/37680 I also tried to reply to that thread but it didn't go through because the old ceph-users@lists.ceph.com mailing list is decommissioned. Thanks, Ilya
Hi Ilya, unfortunately I can not find any "missing primary copy of ..." error in the logs of my 3 OSDs. The NVME disks are also brand new and there is not much traffic on them. The only error keyword I find are those two messages in osd.0 and osd.1 logs shown below. BTW the error posted before actually concerns osd1. The one I posted was copied from somebody elses bug report, which had similar errors. Here are my original error messages on LTSP boot: [ 10.331119] libceph: mon1 (1)10.101.0.27:6789 session established [ 10.331799] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.336866] libceph: mon0 (1)10.101.0.25:6789 session established [ 10.337598] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 *[ 10.349380] libceph: get_reply osd1 tid 11 data 4164 > preallocated* *4096, skipping* elias@maas:~$ juju ssh ceph-osd/2 sudo zgrep -i error /var/log/ceph/ceph-osd.0.log 2021-05-16T08:52:56.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:52:59.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:53:00.884+0000 7f0b262c2d80 1 osd.0 8599 warning: got an error loading one or more classes: (1) Operation not permitted elias@maas:~$ juju ssh ceph-osd/0 sudo zgrep -i error /var/log/ceph/ceph-osd.1.log 2021-05-16T08:49:52.971+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:55.979+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:56.828+0000 7fb6aa68ed80 1 osd.1 8589 warning: got an error loading one or more classes: (1) Operation not permitted How can I find our more about this bug? It keeps coming back every two weeks and I need to restart all OSDs to make it go away for another two weeks. Can I check "tid 11 data 4164" somehow. I find no documentation, what a tid actually is and how I could perform a read test on it. Another interesting detail is, that the problem does only seem to affect booting up from this RBD but not operation per se. The thin clients already booted from this RBD continue working. All systems run: Ubuntu 20.04.2 LTS Kernel 5.8.0-53-generic ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable) The cluster has been setup with Ubuntu MAAS/juju, consists of * 1 MAAS server * with 1 virtual LXD juju controller * 3 OSD servers with one 2 TB Nvme SSD each for ceph and a 256 SATA SSD for the operating system. * each OSD contains a virtualized LXD MON and an LXD FS server (setup through juju, see juju yaml file attached). How can I investigate this problem further or might an upgrade to pacific be necessary? My best regards, Markus Am Mi., 28. Apr. 2021 um 19:25 Uhr schrieb Ilya Dryomov <idryomov@gmail.com
:
On Sun, Apr 25, 2021 at 11:42 AM Ilya Dryomov <idryomov@gmail.com> wrote:
On Sun, Apr 25, 2021 at 12:37 AM Markus Kienast <mark@trickkiste.at>
I am seeing these messages when booting from RBD and booting hangs
wrote: there.
libceph: get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
However, Ceph Health is OK, so I have no idea what is going on. I reboot my 3 node cluster and it works again for about two weeks.
How can I find out more about this issue, how can I dig deeper? Also there has been at least one report about this issue before on this mailing list - "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" - but no solution has been presented.
This report was from 2018, so no idea if this is still an issue for Dyweni the original reporter. If you read this, I would be happy to hear how you solved the problem.
Hi Markus,
What versions of ceph and the kernel are in use?
Are you also seeing I/O errors and "missing primary copy of ..., will try copies on ..." messages in the OSD logs (in this case osd2)?
For the sake of archives, the "[ceph-users] Strange Data Issue - Unexpected client hang on OSD I/O Error" instance has been fixed in 12.2.12, 13.2.5 and 14.2.0:
https://tracker.ceph.com/issues/37680
I also tried to reply to that thread but it didn't go through because the old ceph-users@lists.ceph.com mailing list is decommissioned.
Thanks,
Ilya
On Sun, May 16, 2021 at 12:54 PM Markus Kienast <mark@trickkiste.at> wrote:
Hi Ilya,
unfortunately I can not find any "missing primary copy of ..." error in the logs of my 3 OSDs. The NVME disks are also brand new and there is not much traffic on them.
The only error keyword I find are those two messages in osd.0 and osd.1 logs shown below.
BTW the error posted before actually concerns osd1. The one I posted was copied from somebody elses bug report, which had similar errors. Here are my original error messages on LTSP boot:
Hi Markus, Please don't ever paste log messages from other bug reports again. Your email said "I am seeing these messages ..." and I spent a fair amount of time staring at the code trying to understand how an issue that was fixed several releases ago could resurface. The numbers in the log message mean specific things. For example it is immediately obvious that get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping is not related to get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping even though they probably look the same to you.
[ 10.331119] libceph: mon1 (1)10.101.0.27:6789 session established [ 10.331799] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.336866] libceph: mon0 (1)10.101.0.25:6789 session established [ 10.337598] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.349380] libceph: get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
Please paste the entire boot log and "rbd info" output for the affected image.
elias@maas:~$ juju ssh ceph-osd/2 sudo zgrep -i error /var/log/ceph/ceph-osd.0.log 2021-05-16T08:52:56.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:52:59.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:53:00.884+0000 7f0b262c2d80 1 osd.0 8599 warning: got an error loading one or more classes: (1) Operation not permitted
elias@maas:~$ juju ssh ceph-osd/0 sudo zgrep -i error /var/log/ceph/ceph-osd.1.log 2021-05-16T08:49:52.971+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:55.979+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:56.828+0000 7fb6aa68ed80 1 osd.1 8589 warning: got an error loading one or more classes: (1) Operation not permitted
How can I find our more about this bug? It keeps coming back every two weeks and I need to restart all OSDs to make it go away for another two weeks. Can I check "tid 11 data 4164" somehow. I find no documentation, what a tid actually is and how I could perform a read test on it.
So *just* restarting the three OSDs you have makes it go away? What is meant by restarting? Rebooting the node or simply restarting the OSD process?
Another interesting detail is, that the problem does only seem to affect booting up from this RBD but not operation per se. The thin clients already booted from this RBD continue working.
I take it that the affected image is mapped on multiple nodes? If so, on how many?
All systems run: Ubuntu 20.04.2 LTS Kernel 5.8.0-53-generic ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
The cluster has been setup with Ubuntu MAAS/juju, consists of * 1 MAAS server * with 1 virtual LXD juju controller * 3 OSD servers with one 2 TB Nvme SSD each for ceph and a 256 SATA SSD for the operating system. * each OSD contains a virtualized LXD MON and an LXD FS server (setup through juju, see juju yaml file attached).
Can you describe the client side a bit more? How many clients do you have? How many of them are active at the same time? What exactly is meant by "booting from RBD"? Does the affected image serve as a golden image? Or some other image is snapshotted and cloned from before booting? What do clients do after they boot? Thanks, Ilya
Am So., 16. Mai 2021 um 15:36 Uhr schrieb Ilya Dryomov <idryomov@gmail.com>:
On Sun, May 16, 2021 at 12:54 PM Markus Kienast <mark@trickkiste.at> wrote:
Hi Ilya,
unfortunately I can not find any "missing primary copy of ..." error in
the logs of my 3 OSDs.
The NVME disks are also brand new and there is not much traffic on them.
The only error keyword I find are those two messages in osd.0 and osd.1 logs shown below.
BTW the error posted before actually concerns osd1. The one I posted was copied from somebody elses bug report, which had similar errors. Here are my original error messages on LTSP boot:
Hi Markus,
Please don't ever paste log messages from other bug reports again. Your email said "I am seeing these messages ..." and I spent a fair amount of time staring at the code trying to understand how an issue that was fixed several releases ago could resurface.
The numbers in the log message mean specific things. For example it is immediately obvious that
get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
is not related to
get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
even though they probably look the same to you.
Sorry, I was not aware of that.
[ 10.331119] libceph: mon1 (1)10.101.0.27:6789 session established [ 10.331799] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.336866] libceph: mon0 (1)10.101.0.25:6789 session established [ 10.337598] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.349380] libceph: get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
Please paste the entire boot log and "rbd info" output for the affected image.
elias@maas:~$ rbd info squashfs/ltsp-01 rbd image 'ltsp-01': size 3.5 GiB in 896 objects order 22 (4 MiB objects) snapshot_count: 0 id: 23faade1714 block_name_prefix: rbd_data.23faade1714 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Jan 11 12:09:22 2021 access_timestamp: Wed Feb 24 10:55:17 2021 modify_timestamp: Mon Jan 11 12:09:22 2021 I don't have the boot log available right now, but you can watch a video of the boot process right here: https://photos.app.goo.gl/S8PssYu2VAr4CSeg7 It seems to be consistently "tid 11" consistently, while in this video it was "data 4288" not "data 4164" as above. But the image has been modified in the meantime, as far as I can recall, so that might be due to that reason.
elias@maas:~$ juju ssh ceph-osd/2 sudo zgrep -i error
/var/log/ceph/ceph-osd.0.log
2021-05-16T08:52:56.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:52:59.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:53:00.884+0000 7f0b262c2d80 1 osd.0 8599 warning: got an error loading one or more classes: (1) Operation not permitted
elias@maas:~$ juju ssh ceph-osd/0 sudo zgrep -i error /var/log/ceph/ceph-osd.1.log 2021-05-16T08:49:52.971+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:55.979+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:56.828+0000 7fb6aa68ed80 1 osd.1 8589 warning: got an error loading one or more classes: (1) Operation not permitted
How can I find our more about this bug? It keeps coming back every two weeks and I need to restart all OSDs to make it go away for another two weeks. Can I check "tid 11 data 4164" somehow. I find no documentation, what a tid actually is and how I could perform a read test on it.
So *just* restarting the three OSDs you have makes it go away?
What is meant by restarting? Rebooting the node or simply restarting the OSD process?
I did reboot all OSD nodes and since the MON and FS nodes run as LXD/juju instances on them, they were rebooted as well.
Another interesting detail is, that the problem does only seem to affect
booting up from this RBD but not operation per se. The thin clients already booted from this RBD continue working.
I take it that the affected image is mapped on multiple nodes? If so, on how many?
Currently "squashfs/ltsp-01" is mapped on 4 nodes. As the pool name indicates, the FS was converted to squashfs and is therefore mounted read-only, while the underlying dev might actually not be mounted read-only, as there does not seem to be an option available to mount RO via /sys/bus/rbd/add_single_major or /sys/bus/rbd/add. As far as I can tell, the only way to force RO is to map a snapshot instead.
All systems run: Ubuntu 20.04.2 LTS Kernel 5.8.0-53-generic ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus
(stable)
The cluster has been setup with Ubuntu MAAS/juju, consists of * 1 MAAS server * with 1 virtual LXD juju controller * 3 OSD servers with one 2 TB Nvme SSD each for ceph and a 256 SATA SSD
* each OSD contains a virtualized LXD MON and an LXD FS server (setup
for the operating system. through juju, see juju yaml file attached).
Can you describe the client side a bit more? How many clients do you have? How many of them are active at the same time?
Currently, there are only 4 active clients but the system is intended to being able to sustain 100s of clients. We are using an RBD as boot device for PXE booted thin clients, you might have heard of the Linux Terminal Server Project (ltsp.org). We adapted the stack to support booting from RBD. The clients do not write to the FS/dev, all necessary write operations happen on a mapped overlayfs. Currently we have one virtual machine with an ext4 FS on RBD "rbd/ltsp-01", where we prepare the boot images for LTSP. We generate a squashfs from that FS and then dd it onto a RBD "squashfs/ltsp-01" and back up the previous version by moving it to another RBD name. The initrd mapping process can be reviewed here: https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
What exactly is meant by "booting from RBD"?
Does the affected image serve as a golden image? Or some other image is snapshotted and cloned from before booting?
The same RBD is mapped by all clients and the squashfs image on it is mounted read-only.
What do clients do after they boot?
Normal desktop operation, they serve as fat clients with the RBD as their distributed network boot drive. As said above, all write access is done to an overlayfs ramdisk.
Thanks,
Ilya
Thank you for reviewing this issue. My best regards Markus
On Sun, May 16, 2021 at 4:18 PM Markus Kienast <mark@trickkiste.at> wrote:
Am So., 16. Mai 2021 um 15:36 Uhr schrieb Ilya Dryomov <idryomov@gmail.com>:
On Sun, May 16, 2021 at 12:54 PM Markus Kienast <mark@trickkiste.at> wrote:
Hi Ilya,
unfortunately I can not find any "missing primary copy of ..." error in the logs of my 3 OSDs. The NVME disks are also brand new and there is not much traffic on them.
The only error keyword I find are those two messages in osd.0 and osd.1 logs shown below.
BTW the error posted before actually concerns osd1. The one I posted was copied from somebody elses bug report, which had similar errors. Here are my original error messages on LTSP boot:
Hi Markus,
Please don't ever paste log messages from other bug reports again. Your email said "I am seeing these messages ..." and I spent a fair amount of time staring at the code trying to understand how an issue that was fixed several releases ago could resurface.
The numbers in the log message mean specific things. For example it is immediately obvious that
get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
is not related to
get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
even though they probably look the same to you.
Sorry, I was not aware of that.
[ 10.331119] libceph: mon1 (1)10.101.0.27:6789 session established [ 10.331799] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.336866] libceph: mon0 (1)10.101.0.25:6789 session established [ 10.337598] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.349380] libceph: get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
Please paste the entire boot log and "rbd info" output for the affected image.
elias@maas:~$ rbd info squashfs/ltsp-01 rbd image 'ltsp-01': size 3.5 GiB in 896 objects order 22 (4 MiB objects) snapshot_count: 0 id: 23faade1714 block_name_prefix: rbd_data.23faade1714 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Jan 11 12:09:22 2021 access_timestamp: Wed Feb 24 10:55:17 2021 modify_timestamp: Mon Jan 11 12:09:22 2021
I don't have the boot log available right now, but you can watch a video of the boot process right here: https://photos.app.goo.gl/S8PssYu2VAr4CSeg7
It seems to be consistently "tid 11" consistently, while in this video it was "data 4288" not "data 4164" as above. But the image has been modified in the meantime, as far as I can recall, so that might be due to that reason.
elias@maas:~$ juju ssh ceph-osd/2 sudo zgrep -i error /var/log/ceph/ceph-osd.0.log 2021-05-16T08:52:56.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:52:59.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:53:00.884+0000 7f0b262c2d80 1 osd.0 8599 warning: got an error loading one or more classes: (1) Operation not permitted
elias@maas:~$ juju ssh ceph-osd/0 sudo zgrep -i error /var/log/ceph/ceph-osd.1.log 2021-05-16T08:49:52.971+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:55.979+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:56.828+0000 7fb6aa68ed80 1 osd.1 8589 warning: got an error loading one or more classes: (1) Operation not permitted
How can I find our more about this bug? It keeps coming back every two weeks and I need to restart all OSDs to make it go away for another two weeks. Can I check "tid 11 data 4164" somehow. I find no documentation, what a tid actually is and how I could perform a read test on it.
So *just* restarting the three OSDs you have makes it go away?
What is meant by restarting? Rebooting the node or simply restarting the OSD process?
I did reboot all OSD nodes and since the MON and FS nodes run as LXD/juju instances on them, they were rebooted as well.
Another interesting detail is, that the problem does only seem to affect booting up from this RBD but not operation per se. The thin clients already booted from this RBD continue working.
I take it that the affected image is mapped on multiple nodes? If so, on how many?
Currently "squashfs/ltsp-01" is mapped on 4 nodes. As the pool name indicates, the FS was converted to squashfs and is therefore mounted read-only, while the underlying dev might actually not be mounted read-only, as there does not seem to be an option available to mount RO via /sys/bus/rbd/add_single_major or /sys/bus/rbd/add.
As far as I can tell, the only way to force RO is to map a snapshot instead.
Are you writing to /sys/bus/rbd/add_single_major directly instead of using the rbd tool?
All systems run: Ubuntu 20.04.2 LTS Kernel 5.8.0-53-generic ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
The cluster has been setup with Ubuntu MAAS/juju, consists of * 1 MAAS server * with 1 virtual LXD juju controller * 3 OSD servers with one 2 TB Nvme SSD each for ceph and a 256 SATA SSD for the operating system. * each OSD contains a virtualized LXD MON and an LXD FS server (setup through juju, see juju yaml file attached).
Can you describe the client side a bit more? How many clients do you have? How many of them are active at the same time?
Currently, there are only 4 active clients but the system is intended to being able to sustain 100s of clients. We are using an RBD as boot device for PXE booted thin clients, you might have heard of the Linux Terminal Server Project (ltsp.org). We adapted the stack to support booting from RBD.
How many active clients there were at the time when the image couldn't be mapped? I suspect between 60 and 70? The next time it happens, check the output of "rbd status" for that image. If you see around 65 watchers, that is it. With exclusive-lock feature enabled on the image, the current kernel implementation can't handle more than that. Watches are established if the image is mapped read-write. For your squashfs + overlayfs use case, it's not only better to map read-only just in case, you actually *need* to do that to avoid watches being established. If you are writing to /sys/bus/rbd/add_single_major directly, append "ro" somewhere in the options part of the string: ip:port,... name=myuser,secret=mysecret rbd ltsp-01 - # read-write ip:port,... name=myuser,secret=mysecret,ro rbd ltsp-01 - # read-only Thanks, Ilya
Am So., 16. Mai 2021 um 19:38 Uhr schrieb Ilya Dryomov <idryomov@gmail.com>:
On Sun, May 16, 2021 at 4:18 PM Markus Kienast <mark@trickkiste.at> wrote:
Am So., 16. Mai 2021 um 15:36 Uhr schrieb Ilya Dryomov <
On Sun, May 16, 2021 at 12:54 PM Markus Kienast <mark@trickkiste.at>
wrote:
Hi Ilya,
unfortunately I can not find any "missing primary copy of ..." error
in the logs of my 3 OSDs.
The NVME disks are also brand new and there is not much traffic on
The only error keyword I find are those two messages in osd.0 and
osd.1 logs shown below.
BTW the error posted before actually concerns osd1. The one I posted
was copied from somebody elses bug report, which had similar errors. Here are my original error messages on LTSP boot:
Hi Markus,
Please don't ever paste log messages from other bug reports again. Your email said "I am seeing these messages ..." and I spent a fair amount of time staring at the code trying to understand how an issue that was fixed several releases ago could resurface.
The numbers in the log message mean specific things. For example it is immediately obvious that
get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
is not related to
get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
even though they probably look the same to you.
Sorry, I was not aware of that.
[ 10.331119] libceph: mon1 (1)10.101.0.27:6789 session established [ 10.331799] libceph: client175444 fsid
b0f4a188-bd81-11ea-8849-97abe2843f29
[ 10.336866] libceph: mon0 (1)10.101.0.25:6789 session established [ 10.337598] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.349380] libceph: get_reply osd1 tid 11 data 4164 >
4096, skipping
Please paste the entire boot log and "rbd info" output for the affected image.
elias@maas:~$ rbd info squashfs/ltsp-01 rbd image 'ltsp-01': size 3.5 GiB in 896 objects order 22 (4 MiB objects) snapshot_count: 0 id: 23faade1714 block_name_prefix: rbd_data.23faade1714 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Jan 11 12:09:22 2021 access_timestamp: Wed Feb 24 10:55:17 2021 modify_timestamp: Mon Jan 11 12:09:22 2021
I don't have the boot log available right now, but you can watch a video of the boot process right here: https://photos.app.goo.gl/S8PssYu2VAr4CSeg7
It seems to be consistently "tid 11" consistently, while in this video it was "data 4288" not "data 4164" as above. But the image has been modified in the meantime, as far as I can recall, so that might be due to
elias@maas:~$ juju ssh ceph-osd/2 sudo zgrep -i error
/var/log/ceph/ceph-osd.0.log
2021-05-16T08:52:56.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:52:59.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:53:00.884+0000 7f0b262c2d80 1 osd.0 8599 warning: got an error loading one or more classes: (1) Operation not permitted
elias@maas:~$ juju ssh ceph-osd/0 sudo zgrep -i error /var/log/ceph/ceph-osd.1.log 2021-05-16T08:49:52.971+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:55.979+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:56.828+0000 7fb6aa68ed80 1 osd.1 8589 warning: got an error loading one or more classes: (1) Operation not permitted
How can I find our more about this bug? It keeps coming back every two weeks and I need to restart all OSDs to make it go away for another two weeks. Can I check "tid 11 data 4164" somehow. I find no documentation, what a tid actually is and how I could perform a read test on it.
So *just* restarting the three OSDs you have makes it go away?
What is meant by restarting? Rebooting the node or simply restarting the OSD process?
I did reboot all OSD nodes and since the MON and FS nodes run as LXD/juju instances on them, they were rebooted as well.
Another interesting detail is, that the problem does only seem to
affect booting up from this RBD but not operation per se. The thin clients already booted from this RBD continue working.
I take it that the affected image is mapped on multiple nodes? If so, on how many?
Currently "squashfs/ltsp-01" is mapped on 4 nodes. As the pool name indicates, the FS was converted to squashfs and is
idryomov@gmail.com>: them. preallocated that reason. therefore mounted read-only, while the underlying dev might actually not be mounted read-only, as there does not seem to be an option available to mount RO via /sys/bus/rbd/add_single_major or /sys/bus/rbd/add.
As far as I can tell, the only way to force RO is to map a snapshot
instead.
Are you writing to /sys/bus/rbd/add_single_major directly instead of using the rbd tool?
Yes. Line 110 https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-... echo "${mons} name=${user},secret=${key} ${pool} ${image} ${snap}" > ${rbd_bus}
All systems run: Ubuntu 20.04.2 LTS Kernel 5.8.0-53-generic ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55)
The cluster has been setup with Ubuntu MAAS/juju, consists of * 1 MAAS server * with 1 virtual LXD juju controller * 3 OSD servers with one 2 TB Nvme SSD each for ceph and a 256 SATA
SSD for the operating system.
* each OSD contains a virtualized LXD MON and an LXD FS server (setup
octopus (stable) through juju, see juju yaml file attached).
Can you describe the client side a bit more? How many clients do you have? How many of them are active at the same time?
Currently, there are only 4 active clients but the system is intended to being able to sustain 100s of clients. We are using an RBD as boot device for PXE booted thin clients, you might have heard of the Linux Terminal Server Project (ltsp.org). We adapted the stack to support booting from RBD.
How many active clients there were at the time when the image couldn't be mapped? I suspect between 60 and 70?
No, just 4. Most of the time 3 still running and working correctly and one stuck at reboot. Maybe the sum of all LTSP client reboots since I cleared the problem by rebooting the OSDs could amount to 60-70. I do not know, as we are not logging that currently.
The next time it happens, check the output of "rbd status" for that image. If you see around 65 watchers, that is it. With exclusive-lock feature enabled on the image, the current kernel implementation can't handle more than that.
OK, currently I am seeing 5, which is one more than the number of clients we have. So it seems these watchers do not timeout after reboot or hard reset. Is there any way to make these watchers time out?
Watches are established if the image is mapped read-write. For your squashfs + overlayfs use case, it's not only better to map read-only just in case, you actually *need* to do that to avoid watches being established.
If you are writing to /sys/bus/rbd/add_single_major directly, append "ro" somewhere in the options part of the string:
ip:port,... name=myuser,secret=mysecret rbd ltsp-01 - # read-write
ip:port,... name=myuser,secret=mysecret,ro rbd ltsp-01 - # read-only
Thank you, we will add this missing piece to our rbd initrd code. Are you a ceph dev? Could you make sure to add this to kernel documentation too! https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-rbd There is no mention of that option currently. I might even have tried this but it might not have worked. Not sure, this has been over a year back. Also missing in the documentation is, how one could mount a CephFS on boot!!! We are thinking about switching to booting a CephFS in the future. But I would not have any idea and did not find any documentation on how we would approach that - which boot kernel option to use, which sysfs interface could be used, or which tools we must include in initrd. Generally it would be great if you could include the proper initrd code for RBD and CephFS root filesystems to the Ceph project. You can happily use my code as a starting point. https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-... My best regards Markus
Thanks,
Ilya
On Sun, May 16, 2021 at 8:06 PM Markus Kienast <mark@trickkiste.at> wrote:
Am So., 16. Mai 2021 um 19:38 Uhr schrieb Ilya Dryomov <idryomov@gmail.com>:
On Sun, May 16, 2021 at 4:18 PM Markus Kienast <mark@trickkiste.at> wrote:
Am So., 16. Mai 2021 um 15:36 Uhr schrieb Ilya Dryomov <idryomov@gmail.com>:
On Sun, May 16, 2021 at 12:54 PM Markus Kienast <mark@trickkiste.at> wrote:
Hi Ilya,
unfortunately I can not find any "missing primary copy of ..." error in the logs of my 3 OSDs. The NVME disks are also brand new and there is not much traffic on them.
The only error keyword I find are those two messages in osd.0 and osd.1 logs shown below.
BTW the error posted before actually concerns osd1. The one I posted was copied from somebody elses bug report, which had similar errors. Here are my original error messages on LTSP boot:
Hi Markus,
Please don't ever paste log messages from other bug reports again. Your email said "I am seeing these messages ..." and I spent a fair amount of time staring at the code trying to understand how an issue that was fixed several releases ago could resurface.
The numbers in the log message mean specific things. For example it is immediately obvious that
get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
is not related to
get_reply osd2 tid 1459933 data 3248128 > preallocated 131072, skipping
even though they probably look the same to you.
Sorry, I was not aware of that.
[ 10.331119] libceph: mon1 (1)10.101.0.27:6789 session established [ 10.331799] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.336866] libceph: mon0 (1)10.101.0.25:6789 session established [ 10.337598] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.349380] libceph: get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
Please paste the entire boot log and "rbd info" output for the affected image.
elias@maas:~$ rbd info squashfs/ltsp-01 rbd image 'ltsp-01': size 3.5 GiB in 896 objects order 22 (4 MiB objects) snapshot_count: 0 id: 23faade1714 block_name_prefix: rbd_data.23faade1714 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Jan 11 12:09:22 2021 access_timestamp: Wed Feb 24 10:55:17 2021 modify_timestamp: Mon Jan 11 12:09:22 2021
I don't have the boot log available right now, but you can watch a video of the boot process right here: https://photos.app.goo.gl/S8PssYu2VAr4CSeg7
It seems to be consistently "tid 11" consistently, while in this video it was "data 4288" not "data 4164" as above. But the image has been modified in the meantime, as far as I can recall, so that might be due to that reason.
elias@maas:~$ juju ssh ceph-osd/2 sudo zgrep -i error /var/log/ceph/ceph-osd.0.log 2021-05-16T08:52:56.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:52:59.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:53:00.884+0000 7f0b262c2d80 1 osd.0 8599 warning: got an error loading one or more classes: (1) Operation not permitted
elias@maas:~$ juju ssh ceph-osd/0 sudo zgrep -i error /var/log/ceph/ceph-osd.1.log 2021-05-16T08:49:52.971+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:55.979+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:56.828+0000 7fb6aa68ed80 1 osd.1 8589 warning: got an error loading one or more classes: (1) Operation not permitted
How can I find our more about this bug? It keeps coming back every two weeks and I need to restart all OSDs to make it go away for another two weeks. Can I check "tid 11 data 4164" somehow. I find no documentation, what a tid actually is and how I could perform a read test on it.
So *just* restarting the three OSDs you have makes it go away?
What is meant by restarting? Rebooting the node or simply restarting the OSD process?
I did reboot all OSD nodes and since the MON and FS nodes run as LXD/juju instances on them, they were rebooted as well.
Another interesting detail is, that the problem does only seem to affect booting up from this RBD but not operation per se. The thin clients already booted from this RBD continue working.
I take it that the affected image is mapped on multiple nodes? If so, on how many?
Currently "squashfs/ltsp-01" is mapped on 4 nodes. As the pool name indicates, the FS was converted to squashfs and is therefore mounted read-only, while the underlying dev might actually not be mounted read-only, as there does not seem to be an option available to mount RO via /sys/bus/rbd/add_single_major or /sys/bus/rbd/add.
As far as I can tell, the only way to force RO is to map a snapshot instead.
Are you writing to /sys/bus/rbd/add_single_major directly instead of using the rbd tool?
Yes. Line 110 https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
echo "${mons} name=${user},secret=${key} ${pool} ${image} ${snap}" > ${rbd_bus}
All systems run: Ubuntu 20.04.2 LTS Kernel 5.8.0-53-generic ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)
The cluster has been setup with Ubuntu MAAS/juju, consists of * 1 MAAS server * with 1 virtual LXD juju controller * 3 OSD servers with one 2 TB Nvme SSD each for ceph and a 256 SATA SSD for the operating system. * each OSD contains a virtualized LXD MON and an LXD FS server (setup through juju, see juju yaml file attached).
Can you describe the client side a bit more? How many clients do you have? How many of them are active at the same time?
Currently, there are only 4 active clients but the system is intended to being able to sustain 100s of clients. We are using an RBD as boot device for PXE booted thin clients, you might have heard of the Linux Terminal Server Project (ltsp.org). We adapted the stack to support booting from RBD.
How many active clients there were at the time when the image couldn't be mapped? I suspect between 60 and 70?
No, just 4. Most of the time 3 still running and working correctly and one stuck at reboot.
Maybe the sum of all LTSP client reboots since I cleared the problem by rebooting the OSDs could amount to 60-70. I do not know, as we are not logging that currently.
The next time it happens, check the output of "rbd status" for that image. If you see around 65 watchers, that is it. With exclusive-lock feature enabled on the image, the current kernel implementation can't handle more than that.
OK, currently I am seeing 5, which is one more than the number of clients we have. So it seems these watchers do not timeout after reboot or hard reset.
Is there any way to make these watchers time out?
They are supposed to time out after 30 seconds. Does the IP address of the rogue watch offer a clue? Note that when the mapping gets stuck on that preallocated check, it still maintains the watch so it's not going to time out in that case.
Watches are established if the image is mapped read-write. For your squashfs + overlayfs use case, it's not only better to map read-only just in case, you actually *need* to do that to avoid watches being established.
If you are writing to /sys/bus/rbd/add_single_major directly, append "ro" somewhere in the options part of the string:
ip:port,... name=myuser,secret=mysecret rbd ltsp-01 - # read-write
ip:port,... name=myuser,secret=mysecret,ro rbd ltsp-01 - # read-only
Thank you, we will add this missing piece to our rbd initrd code.
Are you a ceph dev? Could you make sure to add this to kernel documentation too! https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-rbd
Map options are documented in the rbd man page: https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
There is no mention of that option currently. I might even have tried this but it might not have worked. Not sure, this has been over a year back.
Also missing in the documentation is, how one could mount a CephFS on boot!!!
Do you mean booting *from* CephFS, i.e. using it as a root filesystem? Because mounting CephFS on boot after root filesystem is mounted is done through /etc/fstab, like you would mount any other filesystem whether local or network.
We are thinking about switching to booting a CephFS in the future. But I would not have any idea and did not find any documentation on how we would approach that - which boot kernel option to use, which sysfs interface could be used, or which tools we must include in initrd.
Generally it would be great if you could include the proper initrd code for RBD and CephFS root filesystems to the Ceph project. You can happily use my code as a starting point.
https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
I think booting from CephFS would require kernel patches. It looks like NFS and CIFS are the only network filesystems supported by the init/root infrastructure in the kernel. Thanks, Ilya
Am So., 16. Mai 2021 um 21:36 Uhr schrieb Ilya Dryomov <idryomov@gmail.com>:
On Sun, May 16, 2021 at 8:06 PM Markus Kienast <mark@trickkiste.at> wrote:
Am So., 16. Mai 2021 um 19:38 Uhr schrieb Ilya Dryomov <
On Sun, May 16, 2021 at 4:18 PM Markus Kienast <mark@trickkiste.at>
wrote:
Am So., 16. Mai 2021 um 15:36 Uhr schrieb Ilya Dryomov <
idryomov@gmail.com>:
On Sun, May 16, 2021 at 12:54 PM Markus Kienast <mark@trickkiste.at>
wrote:
Hi Ilya,
unfortunately I can not find any "missing primary copy of ..."
error in the logs of my 3 OSDs.
The NVME disks are also brand new and there is not much traffic on
The only error keyword I find are those two messages in osd.0 and
osd.1 logs shown below.
BTW the error posted before actually concerns osd1. The one I
Hi Markus,
Please don't ever paste log messages from other bug reports again. Your email said "I am seeing these messages ..." and I spent a fair amount of time staring at the code trying to understand how an issue that was fixed several releases ago could resurface.
The numbers in the log message mean specific things. For example it is immediately obvious that
get_reply osd1 tid 11 data 4164 > preallocated 4096, skipping
is not related to
get_reply osd2 tid 1459933 data 3248128 > preallocated 131072,
skipping
even though they probably look the same to you.
Sorry, I was not aware of that.
[ 10.331119] libceph: mon1 (1)10.101.0.27:6789 session
established
[ 10.331799] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.336866] libceph: mon0 (1)10.101.0.25:6789 session established [ 10.337598] libceph: client175444 fsid b0f4a188-bd81-11ea-8849-97abe2843f29 [ 10.349380] libceph: get_reply osd1 tid 11 data 4164 >
4096, skipping
Please paste the entire boot log and "rbd info" output for the affected image.
elias@maas:~$ rbd info squashfs/ltsp-01 rbd image 'ltsp-01': size 3.5 GiB in 896 objects order 22 (4 MiB objects) snapshot_count: 0 id: 23faade1714 block_name_prefix: rbd_data.23faade1714 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Jan 11 12:09:22 2021 access_timestamp: Wed Feb 24 10:55:17 2021 modify_timestamp: Mon Jan 11 12:09:22 2021
I don't have the boot log available right now, but you can watch a video of the boot process right here: https://photos.app.goo.gl/S8PssYu2VAr4CSeg7
It seems to be consistently "tid 11" consistently, while in this video it was "data 4288" not "data 4164" as above. But the image has been modified in the meantime, as far as I can recall, so that might be due to
elias@maas:~$ juju ssh ceph-osd/2 sudo zgrep -i error
/var/log/ceph/ceph-osd.0.log
2021-05-16T08:52:56.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:52:59.872+0000 7f0b262c2d80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:53:00.884+0000 7f0b262c2d80 1 osd.0 8599 warning: got an error loading one or more classes: (1) Operation not permitted
elias@maas:~$ juju ssh ceph-osd/0 sudo zgrep -i error /var/log/ceph/ceph-osd.1.log 2021-05-16T08:49:52.971+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:55.979+0000 7fb6aa68ed80 4 rocksdb: Options.error_if_exists: 0 2021-05-16T08:49:56.828+0000 7fb6aa68ed80 1 osd.1 8589 warning: got an error loading one or more classes: (1) Operation not permitted
How can I find our more about this bug? It keeps coming back every two weeks and I need to restart all OSDs to make it go away for another two weeks. Can I check "tid 11 data 4164" somehow. I find no documentation, what a tid actually is and how I could perform a read test on it.
So *just* restarting the three OSDs you have makes it go away?
What is meant by restarting? Rebooting the node or simply restarting the OSD process?
I did reboot all OSD nodes and since the MON and FS nodes run as LXD/juju instances on them, they were rebooted as well.
Another interesting detail is, that the problem does only seem to
affect booting up from this RBD but not operation per se. The thin clients already booted from this RBD continue working.
I take it that the affected image is mapped on multiple nodes? If
so,
on how many?
Currently "squashfs/ltsp-01" is mapped on 4 nodes. As the pool name indicates, the FS was converted to squashfs and is
idryomov@gmail.com>: them. posted was copied from somebody elses bug report, which had similar errors. Here are my original error messages on LTSP boot: preallocated that reason. therefore mounted read-only, while the underlying dev might actually not be mounted read-only, as there does not seem to be an option available to mount RO via /sys/bus/rbd/add_single_major or /sys/bus/rbd/add.
As far as I can tell, the only way to force RO is to map a snapshot
instead.
Are you writing to /sys/bus/rbd/add_single_major directly instead of using the rbd tool?
Yes. Line 110 https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
echo "${mons} name=${user},secret=${key} ${pool} ${image} ${snap}" > ${rbd_bus}
All systems run: Ubuntu 20.04.2 LTS Kernel 5.8.0-53-generic ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55)
octopus (stable)
The cluster has been setup with Ubuntu MAAS/juju, consists of * 1 MAAS server * with 1 virtual LXD juju controller * 3 OSD servers with one 2 TB Nvme SSD each for ceph and a 256
SATA SSD for the operating system.
* each OSD contains a virtualized LXD MON and an LXD FS server (setup through juju, see juju yaml file attached).
Can you describe the client side a bit more? How many clients do you have? How many of them are active at the same time?
Currently, there are only 4 active clients but the system is intended to being able to sustain 100s of clients. We are using an RBD as boot device for PXE booted thin clients, you might have heard of the Linux Terminal Server Project (ltsp.org). We adapted the stack to support booting from RBD.
How many active clients there were at the time when the image couldn't be mapped? I suspect between 60 and 70?
No, just 4. Most of the time 3 still running and working correctly and one stuck at reboot.
Maybe the sum of all LTSP client reboots since I cleared the problem by rebooting the OSDs could amount to 60-70. I do not know, as we are not logging that currently.
The next time it happens, check the output of "rbd status" for that image. If you see around 65 watchers, that is it. With exclusive-lock feature enabled on the image, the current kernel implementation can't handle more than that.
OK, currently I am seeing 5, which is one more than the number of clients we have. So it seems these watchers do not timeout after reboot or hard reset.
Is there any way to make these watchers time out?
They are supposed to time out after 30 seconds. Does the IP address of the rogue watch offer a clue?
Not really, I see 3 watchers for one of my client IPs right now and one for each of the others. Have to investigate further. The clients are assigned the same IP on each bootup, so shouldn't the watchers either time out or be "claimed/taken over" by the newly booted client?
Note that when the mapping gets stuck on that preallocated check, it still maintains the watch so it's not going to time out in that case.
I will investigate, if it is only one client, producing all the watchers until the limit is reached, or if all clients produce this problem together and then report back on the issue.
Watches are established if the image is mapped read-write. For your squashfs + overlayfs use case, it's not only better to map read-only just in case, you actually *need* to do that to avoid watches being established.
If you are writing to /sys/bus/rbd/add_single_major directly, append "ro" somewhere in the options part of the string:
ip:port,... name=myuser,secret=mysecret rbd ltsp-01 - # read-write
ip:port,... name=myuser,secret=mysecret,ro rbd ltsp-01 - # read-only
Thank you, we will add this missing piece to our rbd initrd code.
Are you a ceph dev? Could you make sure to add this to kernel documentation too! https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-rbd
Map options are documented in the rbd man page:
https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options
Thanks for the tip! However, I would argue that these krbd options should either be available in the kernel documentation as well, or at least this man page should be mentioned in the kernel documentation.
There is no mention of that option currently. I might even have tried this but it might not have worked. Not sure,
this has been over a year back.
Also missing in the documentation is, how one could mount a CephFS on
boot!!!
Do you mean booting *from* CephFS, i.e. using it as a root filesystem? Because mounting CephFS on boot after root filesystem is mounted is done through /etc/fstab, like you would mount any other filesystem whether local or network.
Yes, CephFS as a roof filesystem. The reason, why this would be convenient is, that we could then have one VM or client mounted this CephFS in RW mode, therefore being able to make updates to the packages installed etc. while the other clients would be restricted to read-only rights via authx. We would still be able to create snapshots of the last properly working OS configuration but without the additional hassle. Also, if we do OS package upgrades, these upgrades would be instantly available on the clients as well without the need to reboot, since due to the network-fs nature of CephFS all clients would be aware of the underlying changes to the FS.
We are thinking about switching to booting a CephFS in the future. But I would not have any idea and did not find any documentation on how we would approach that - which boot kernel option to use, which sysfs interface could be used, or which tools we must include in initrd.
Generally it would be great if you could include the proper initrd code for RBD and CephFS root filesystems to the Ceph project. You can happily use my code as a starting point.
https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
I think booting from CephFS would require kernel patches. It looks like NFS and CIFS are the only network filesystems supported by the init/root infrastructure in the kernel.
As long as we can do it similar to the way we do with RBD now, that would be sufficient. The actual mapping and mounting in both NFS and RBD does take place in the initrd scripts. The RBD initrd script actually is derived from the NFS initrd script, so I would presume that one could make CephFS mount work in a similar fashion. Question being, which tools are needed to make this work? I guess "mount" provides everything needed already, does it not?
Thank you so much for you support, Markus Thanks,
Ilya
Hey Markus, Ilya, you don't know with how much interest I am following this thread, because ...
Generally it would be great if you could include the proper initrd code for RBD and CephFS root filesystems to the Ceph project. You can happily use my code as a starting point.
https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
I think booting from CephFS would require kernel patches. It looks like NFS and CIFS are the only network filesystems supported by the init/root infrastructure in the kernel.
... we have been looking for a while to a discussion about using RBD (not cephfs) as a replacement for a hard disk. Linux can map RBD devices, so should Linux not also be able to *boot* from an rbd device similar to a regular disk? I did not find any example of this yet, but I'd assume that conceptually one would probably: - preload a Linux kernel from the network (potentially via ipxe) - specify root=rbd://fsid/pool/image Or in a even *better* variant: - the bootloader (ipxe?) can map RBD - the bootloader pre-loads enough of the image for reading the partition - the bootloader either loads the kernel + initramfs *or* chainloads another bootloader What are your thoughts on this? Do-able or totally crazy? Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch
Hi Nico, we are already doing exactly that: Loading initrd via iPXE which contains the necessary modules and scripts to boot an RBD boot dev. Works just fine. And Ilya just helped to work out the last show stopper, thanks again for that! We are using a modified LTSP system for this. We have proposed some patches to LTSP to get the necessary facilities upstream but Alkis Georgopoulos first want to see that there is enough interest for that before he considers merging our patch or creating the necessary changes himself. However the necessary initrd code is already available in this merge request: https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-... I see you are from Switzerland - neighbors! Out of interest, what are you planning to use this for? Servers, Thin/Fat Clients? My best regards Markus Am So., 16. Mai 2021 um 23:18 Uhr schrieb Nico Schottelius < nico.schottelius@ungleich.ch>:
Hey Markus, Ilya,
you don't know with how much interest I am following this thread, because ...
Generally it would be great if you could include the proper initrd code for RBD and CephFS root filesystems to the Ceph project. You can happily use my code as a starting point.
https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
I think booting from CephFS would require kernel patches. It looks like NFS and CIFS are the only network filesystems supported by the init/root infrastructure in the kernel.
... we have been looking for a while to a discussion about using RBD (not cephfs) as a replacement for a hard disk. Linux can map RBD devices, so should Linux not also be able to *boot* from an rbd device similar to a regular disk?
I did not find any example of this yet, but I'd assume that conceptually one would probably:
- preload a Linux kernel from the network (potentially via ipxe) - specify root=rbd://fsid/pool/image
Or in a even *better* variant:
- the bootloader (ipxe?) can map RBD - the bootloader pre-loads enough of the image for reading the partition - the bootloader either loads the kernel + initramfs *or* chainloads another bootloader
What are your thoughts on this? Do-able or totally crazy?
Best regards,
Nico
-- Sustainable and modern Infrastructures by ungleich.ch
Markus Kienast <mark@trickkiste.at> writes:
Hi Nico,
we are already doing exactly that:
Loading initrd via iPXE which contains the necessary modules and scripts to boot an RBD boot dev. Works just fine.
Interesting and very good to hear. How do you handle kernel differences (loaded kernel vs. modules in the RBD image)?
And Ilya just helped to work out the last show stopper, thanks again for that!
We are using a modified LTSP system for this.
We have proposed some patches to LTSP to get the necessary facilities upstream but Alkis Georgopoulos first want to see that there is enough interest for that before he considers merging our patch or creating the necessary changes himself.
I think seeing LTSP booting on RBD is a great move forward, also for other projects.
However the necessary initrd code is already available in this merge request: https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
I see you are from Switzerland - neighbors!
We might actually meet at a Linuxtag - but we should probably take this off-list :-)
Out of interest, what are you planning to use this for? Servers, Thin/Fat Clients?
Our objective in the end is to boot servers and VMs from possible the same RBD pool/images. The problem there is though that we don't know what is inside the RBD image, so we don't know which kernel to load besides we would do some kind of kexec magic, which would pass on the RBD parameters. Or in other words, we have this use case: - a customer books a VM and needs more performance - the customer decides to go with *a* server, but not necessarily a specific server - the customer VM should be shutdown and the RBD image should boot on a server If the server crashes, the OS should be booted on a different server. We can obviously work around this by *always* running a VM, but this is not exactly what our customers want. At the moment they use local disks + nfs shares to achieve a similar solution, but it is far from perfect. Cheers, Nico -- Sustainable and modern Infrastructures by ungleich.ch
Am Mo., 17. Mai 2021 um 20:28 Uhr schrieb Nico Schottelius < nico.schottelius@ungleich.ch>:
Markus Kienast <mark@trickkiste.at> writes:
Hi Nico,
we are already doing exactly that:
Loading initrd via iPXE which contains the necessary modules and scripts to boot an RBD boot dev. Works just fine.
Interesting and very good to hear. How do you handle kernel differences (loaded kernel vs. modules in the RBD image)?
The way it works in LTSP is, that you run a script provided by the ltsp project, which copies the kernel to your tftpboot directory and sets the ipxe boot options accordingly. Currently there is no automatism for mapping and mounting the RBD for this purpose, you have to map it by hand and mount it to the proper dir prior to running the script. I am sure you can integrate this into your workflow by using what is currently in ltsp and my rbd branch as an example.
And Ilya just helped to work out the last show stopper, thanks again for that!
We are using a modified LTSP system for this.
We have proposed some patches to LTSP to get the necessary facilities upstream but Alkis Georgopoulos first want to see that there is enough interest for that before he considers merging our patch or creating the necessary changes himself.
I think seeing LTSP booting on RBD is a great move forward, also for other projects.
However the necessary initrd code is already available in this merge request:
https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-...
I see you are from Switzerland - neighbors!
We might actually meet at a Linuxtag - but we should probably take this off-list :-)
In Vienna? Not been there for ages but happy to meet.
Out of interest, what are you planning to use this for? Servers, Thin/Fat Clients?
Our objective in the end is to boot servers and VMs from possible the same RBD pool/images.
The problem there is though that we don't know what is inside the RBD image, so we don't know which kernel to load besides we would do some kind of kexec magic, which would pass on the RBD parameters.
Or in other words, we have this use case:
- a customer books a VM and needs more performance - the customer decides to go with *a* server, but not necessarily a specific server - the customer VM should be shutdown and the RBD image should boot on a server
If the server crashes, the OS should be booted on a different server.
We can obviously work around this by *always* running a VM, but this is not exactly what our customers want. At the moment they use local disks + nfs shares to achieve a similar solution, but it is far from perfect.
Alright, I understand. Yes, your scenario would work, while you would only need to borrow the ipxe stuff from LTSP and my rbd initramfs-hook and initramfs-script. The initramfs stuff should actually be molded into a separate Ubuntu package and made available to upstream. The mapping RO stuff, which Ilya has provided yet needs to be added and all the other available kernel options should be made available as well. My best regards Markus
Cheers,
Nico
-- Sustainable and modern Infrastructures by ungleich.ch
Hi, This is a chicken and egg problem I guess. The boot process (albeit UEFI or BIOS; given x86) should be able to load boot loader code, a Linux kernel and initial RAM disk (although in some cases a kernel alone could be enough). So yes: use PXE to load a Linux kernel and RAM disk. The RAM disk should include RBD kernel modules and a hook to map and mount the RBD device. The code referred to seems it could do the trick. But that's a real world scenario that is usable right now. In terms of crazy ideas: implement an iSCSI like UEFI module supporting RBD acting as a software based HBA. Or even cooler: a hardware HBA similar to Coraid's adapters made for AoE. Cheers, Kees On 16-05-2021 23:19, Nico Schottelius wrote:
Hey Markus, Ilya,
you don't know with how much interest I am following this thread, because ...
Generally it would be great if you could include the proper initrd code for RBD and CephFS root filesystems to the Ceph project. You can happily use my code as a starting point.
https://github.com/trickkiste/ltsp/blob/feature-boot_method-rbd/debian/ltsp-... I think booting from CephFS would require kernel patches. It looks like NFS and CIFS are the only network filesystems supported by the init/root infrastructure in the kernel. ... we have been looking for a while to a discussion about using RBD (not cephfs) as a replacement for a hard disk. Linux can map RBD devices, so should Linux not also be able to *boot* from an rbd device similar to a regular disk?
I did not find any example of this yet, but I'd assume that conceptually one would probably:
- preload a Linux kernel from the network (potentially via ipxe) - specify root=rbd://fsid/pool/image
Or in a even *better* variant:
- the bootloader (ipxe?) can map RBD - the bootloader pre-loads enough of the image for reading the partition - the bootloader either loads the kernel + initramfs *or* chainloads another bootloader
What are your thoughts on this? Do-able or totally crazy?
Best regards,
Nico
participants (4)
-
Ilya Dryomov
-
Kees Meijs | Nefos
-
Markus Kienast
-
Nico Schottelius