Upgrade from Luminous to Nautilus 14.2.9 RBD issue?
Hi, We are experiencing a weird issue after upgrading our clusters from ceph luminous to nautilus 14.2.9 - I am not even sure if this is ceph related but this started to happen exactly after we upgraded, so, I am trying my luck here. We have one ceph rbd pool size 3 min size 2 from all bluestore osds (KRBD) I will try to be clear enough.. though I cannot understand exactly whats happening or whats causing the issue. So, we have 1 virtual machine which uses a rbd image of 2TB - virtio-scsi device. Inside the VM we are trying to create ploop devices to be used for/by containers(inside the VM on the 2TB rbd image QEMU DISK). There is no way we can create ploop devices, it always crash, please check the crash below: https://pastebin.com/9khp9XS3 - sdb in the crash is the 2TB rbd image which the VM uses. There are no other read/write errors, we have health_ok, all OSDs are fine, no errors on any of the phisical disks - this happens only when we want to create ploop devices inside a VM and right after we upgraded our cluster to nautilus 14.2.9. I also did new images/other hosts.. same result. Did try a lot of different versions of ploop packages, same result. I would appreciate if someone else has encountered something similar and if there is a workaround. -- Best Regards, ------------------------------------------------------------------------ Daniel Stan Senior System Administrator | NAV Communications (RO) Office: +40 (21) 655-55-55 | E-Mail: daniel@nav.ro Site: www.nav.ro <https://www.nav.ro> | Client: https://client.ro
On Wed, Jul 1, 2020 at 3:23 AM Daniel Stan - nav.ro <daniel@nav.ro> wrote:
Hi,
We are experiencing a weird issue after upgrading our clusters from ceph luminous to nautilus 14.2.9 - I am not even sure if this is ceph related but this started to happen exactly after we upgraded, so, I am trying my luck here.
We have one ceph rbd pool size 3 min size 2 from all bluestore osds (KRBD)
Are you using krbd or librbd via QEMU?
I will try to be clear enough.. though I cannot understand exactly whats happening or whats causing the issue.
So, we have 1 virtual machine which uses a rbd image of 2TB - virtio-scsi device. Inside the VM we are trying to create ploop devices to be used for/by containers(inside the VM on the 2TB rbd image QEMU DISK).
There is no way we can create ploop devices, it always crash, please check the crash below:
https://pastebin.com/9khp9XS3 - sdb in the crash is the 2TB rbd image which the VM uses.
This sounds like a permissions issue for blacklisting dead clients [1]
There are no other read/write errors, we have health_ok, all OSDs are fine, no errors on any of the phisical disks - this happens only when we want to create ploop devices inside a VM and right after we upgraded our cluster to nautilus 14.2.9. I also did new images/other hosts.. same result. Did try a lot of different versions of ploop packages, same result.
I would appreciate if someone else has encountered something similar and if there is a workaround.
--
Best Regards,
------------------------------------------------------------------------
Daniel Stan
Senior System Administrator | NAV Communications (RO)
Office: +40 (21) 655-55-55 | E-Mail: daniel@nav.ro
Site: www.nav.ro <https://www.nav.ro> | Client: https://client.ro
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-July/027862.html -- Jason
Hi Are there any 'official' or even 'works for us' pointers to 'tuned profiles' for such common uses as 'ceph baremetal osd host' 'ceph osd + libvirt host' 'ceph mon/mgr' 'guest vm based on a kernel-mounted rbd' 'guest vm based on a direct virtio->rados link' I suppose there are a few other common configurations, but you get the idea. If you haven't used or know of 'tuned'-- it's a nice way to collect a great whole lot of sysctl and other low level configuration options in one spot. https://tuned-project.org/ Thanks Harry Coin
Just curious, how does the libvirt xml part look like of a 'direct virtio->rados link' and 'kernel-mounted rbd' -----Original Message----- To: ceph-users@ceph.io Subject: *****SPAM***** [ceph-users] Are there 'tuned profiles' for various ceph scenarios? Hi Are there any 'official' or even 'works for us' pointers to 'tuned profiles' for such common uses as 'ceph baremetal osd host' 'ceph osd + libvirt host' 'ceph mon/mgr' 'guest vm based on a kernel-mounted rbd' 'guest vm based on a direct virtio->rados link' I suppose there are a few other common configurations, but you get the idea. If you haven't used or know of 'tuned'-- it's a nice way to collect a great whole lot of sysctl and other low level configuration options in one spot. https://tuned-project.org/ Thanks Harry Coin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Marc: Here's a template that works here. You'll need to do some steps to create the 'secret' and make the block devs and so on: <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writeback"/> <auth username="libvirt"> <secret type="ceph" uuid="3fc5de60-358e-e010-93c9-bacfc9636edf"/> </auth> <source protocol="rbd" name="yournamespace/yourblockdevname" index="1"> <host name="yourmon1" port="6789"/> <host name="yourmon2" port="6789"/> <host name="yourmon3" port="6789"/> <host name="yourmon4" port="6789"/> </source> <target dev="vda" bus="virtio"/> <alias name="virtio-disk0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/> </disk> Glad I could contribute something. Sure would appreciate leads for the suggested sysctls/etc either apart or as tuned profiles. Harry On 7/1/20 2:44 PM, Marc Roos wrote:
Just curious, how does the libvirt xml part look like of a 'direct virtio->rados link' and 'kernel-mounted rbd'
-----Original Message----- To: ceph-users@ceph.io Subject: *****SPAM***** [ceph-users] Are there 'tuned profiles' for various ceph scenarios?
Hi
Are there any 'official' or even 'works for us' pointers to 'tuned profiles' for such common uses as
'ceph baremetal osd host'
'ceph osd + libvirt host'
'ceph mon/mgr'
'guest vm based on a kernel-mounted rbd'
'guest vm based on a direct virtio->rados link'
I suppose there are a few other common configurations, but you get the idea.
If you haven't used or know of 'tuned'-- it's a nice way to collect a great whole lot of sysctl and other low level configuration options in one spot. https://tuned-project.org/
Thanks
Harry Coin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[Resent to correct title] Marc: Here's a template that works here. You'll need to do some steps to create the 'secret' and make the block devs and so on: <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writeback"/> <auth username="libvirt"> <secret type="ceph" uuid="3fc5de60-358e-e010-93c9-bacfc9636edf"/> </auth> <source protocol="rbd" name="yournamespace/yourblockdevname" index="1"> <host name="yourmon1" port="6789"/> <host name="yourmon2" port="6789"/> <host name="yourmon3" port="6789"/> <host name="yourmon4" port="6789"/> </source> <target dev="vda" bus="virtio"/> <alias name="virtio-disk0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/> </disk> Glad I could contribute something. Sure would appreciate leads for the suggested sysctls/etc either apart or as tuned profiles. Harry On 7/1/20 2:44 PM, Marc Roos wrote:
Just curious, how does the libvirt xml part look like of a 'direct virtio->rados link' and 'kernel-mounted rbd'
-----Original Message----- To: ceph-users@ceph.io Subject: *****SPAM***** [ceph-users] Are there 'tuned profiles' for various ceph scenarios?
Hi
Are there any 'official' or even 'works for us' pointers to 'tuned profiles' for such common uses as
'ceph baremetal osd host'
'ceph osd + libvirt host'
'ceph mon/mgr'
'guest vm based on a kernel-mounted rbd'
'guest vm based on a direct virtio->rados link'
I suppose there are a few other common configurations, but you get the idea.
If you haven't used or know of 'tuned'-- it's a nice way to collect a great whole lot of sysctl and other low level configuration options in one spot. https://tuned-project.org/
Thanks
Harry Coin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Here's an example for SCSI disks (the main benefit vs VirtIO is discard/unmap/TRIM support): <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' io='threads' discard='unmap'/> <auth username='libvirt'> <secret type='ceph' uuid='3fc5de60-358e-e010-93c9-bacfc9636edf'/> </auth> <source protocol='rbd' name='yourpool/yourrbd'> <host name='yourmon1' port='6789'/> <host name='yourmon2' port='6789'/> <host name='yourmon3' port='6789'/> </source> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> You also need a VirtIO-SCSI controller to use these, which will look something like: <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> Cheers, Simon. On 01/07/2020 20:52, Harry G. Coin wrote:
[Resent to correct title] Marc:
Here's a template that works here. You'll need to do some steps to create the 'secret' and make the block devs and so on:
<disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writeback"/> <auth username="libvirt"> <secret type="ceph" uuid="3fc5de60-358e-e010-93c9-bacfc9636edf"/> </auth> <source protocol="rbd" name="yournamespace/yourblockdevname" index="1"> <host name="yourmon1" port="6789"/> <host name="yourmon2" port="6789"/> <host name="yourmon3" port="6789"/> <host name="yourmon4" port="6789"/> </source> <target dev="vda" bus="virtio"/> <alias name="virtio-disk0"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/> </disk>
Glad I could contribute something. Sure would appreciate leads for the suggested sysctls/etc either apart or as tuned profiles.
Harry
On 7/1/20 2:44 PM, Marc Roos wrote:
Just curious, how does the libvirt xml part look like of a 'direct virtio->rados link' and 'kernel-mounted rbd'
-----Original Message----- To: ceph-users@ceph.io Subject: *****SPAM***** [ceph-users] Are there 'tuned profiles' for various ceph scenarios?
Hi
Are there any 'official' or even 'works for us' pointers to 'tuned profiles' for such common uses as
'ceph baremetal osd host'
'ceph osd + libvirt host'
'ceph mon/mgr'
'guest vm based on a kernel-mounted rbd'
'guest vm based on a direct virtio->rados link'
I suppose there are a few other common configurations, but you get the idea.
If you haven't used or know of 'tuned'-- it's a nice way to collect a great whole lot of sysctl and other low level configuration options in one spot. https://tuned-project.org/
Thanks
Harry Coin
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You can find things on the net. Here one starting point: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-May/034890.html Tuning depends heavily on way too many parameters, so generally applicable templates are probably non-existent. Every hardware has its own challenges. If you want to tune properly, I don't see a way around reading the entire kernel tuning parameter references. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Harry G. Coin <hgcoin@gmail.com> Sent: 01 July 2020 21:26:59 To: ceph-users@ceph.io Subject: [ceph-users] Are there 'tuned profiles' for various ceph scenarios? Hi Are there any 'official' or even 'works for us' pointers to 'tuned profiles' for such common uses as 'ceph baremetal osd host' 'ceph osd + libvirt host' 'ceph mon/mgr' 'guest vm based on a kernel-mounted rbd' 'guest vm based on a direct virtio->rados link' I suppose there are a few other common configurations, but you get the idea. If you haven't used or know of 'tuned'-- it's a nice way to collect a great whole lot of sysctl and other low level configuration options in one spot. https://tuned-project.org/ Thanks Harry Coin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, We use KRBD. Our ceph nautilus storage cluster is used for our proxmox 6.x cloud VM disks. This is a bit different as we are not experiencing any errors during boots/reboots we had no unplanned outage, etc., we just safely upgraded from luminous to nautilus and we only experience this when we try to create openvz ploops. I have checked the list but I cant seem to find a solution here. So, we are running nautilus 14.2.9. You can deploy on a VM: https://download.openvz.org/virtuozzo/releases/openvz-7.0.14-136/x86_64/iso/... And just try to create an openvz template cache which will try to add a ploop. vzpkg install template centos-7-x86_64 vzpkg create cache centos-7-x86_64 Or check complete usage: https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_command_line_reference/... Wrote that here as I thought maybe someone could try reproducing this. Best Regards, ------------------------------------------------------------------------ Daniel Stan Senior System Administrator | NAV Communications (RO) Office: +40 (21) 655-55-55 | E-Mail: daniel@nav.ro Site: www.nav.ro <https://www.nav.ro> | Client: https://client.ro On 01/07/2020 15:23, Jason Dillaman wrote:
On Wed, Jul 1, 2020 at 3:23 AM Daniel Stan - nav.ro <daniel@nav.ro> wrote:
Hi,
We are experiencing a weird issue after upgrading our clusters from ceph luminous to nautilus 14.2.9 - I am not even sure if this is ceph related but this started to happen exactly after we upgraded, so, I am trying my luck here.
We have one ceph rbd pool size 3 min size 2 from all bluestore osds (KRBD) Are you using krbd or librbd via QEMU?
I will try to be clear enough.. though I cannot understand exactly whats happening or whats causing the issue.
So, we have 1 virtual machine which uses a rbd image of 2TB - virtio-scsi device. Inside the VM we are trying to create ploop devices to be used for/by containers(inside the VM on the 2TB rbd image QEMU DISK).
There is no way we can create ploop devices, it always crash, please check the crash below:
https://pastebin.com/9khp9XS3 - sdb in the crash is the 2TB rbd image which the VM uses. This sounds like a permissions issue for blacklisting dead clients [1]
There are no other read/write errors, we have health_ok, all OSDs are fine, no errors on any of the phisical disks - this happens only when we want to create ploop devices inside a VM and right after we upgraded our cluster to nautilus 14.2.9. I also did new images/other hosts.. same result. Did try a lot of different versions of ploop packages, same result.
I would appreciate if someone else has encountered something similar and if there is a workaround.
--
Best Regards,
------------------------------------------------------------------------
Daniel Stan
Senior System Administrator | NAV Communications (RO)
Office: +40 (21) 655-55-55 | E-Mail: daniel@nav.ro
Site: www.nav.ro <https://www.nav.ro> | Client: https://client.ro
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-July/027862.html
participants (6)
-
Daniel Stan - nav.ro
-
Frank Schilder
-
Harry G. Coin
-
Jason Dillaman
-
Marc Roos
-
Simon Ironside