Hi Are there any 'official' or even 'works for us' pointers to 'tuned profiles' for such common uses as 'ceph baremetal osd host' 'ceph osd + libvirt host' 'ceph mon/mgr' 'guest vm based on a kernel-mounted rbd' 'guest vm based on a direct virtio->rados link' I suppose there are a few other common configurations, but you get the idea. If you haven't used or know of 'tuned'-- it's a nice way to collect a great whole lot of sysctl and other low level configuration options in one spot. https://tuned-project.org/ Thanks Harry Coin

Marc: Here's a template that works here.  You'll need to do some steps to create the 'secret' and make the block devs and so on: <disk type="network" device="disk">   <driver name="qemu" type="raw" cache="writeback"/>   <auth username="libvirt">     <secret type="ceph" uuid="3fc5de60-358e-e010-93c9-bacfc9636edf"/>   </auth>   <source protocol="rbd" name="yournamespace/yourblockdevname" index="1">     <host name="yourmon1" port="6789"/>     <host name="yourmon2" port="6789"/>     <host name="yourmon3" port="6789"/>     <host name="yourmon4" port="6789"/>   </source>   <target dev="vda" bus="virtio"/>   <alias name="virtio-disk0"/>   <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/> </disk> Glad I could contribute something.   Sure would appreciate leads for the suggested sysctls/etc either apart or as tuned profiles. Harry On 7/1/20 2:44 PM, Marc Roos wrote: > > Just curious, how does the libvirt xml part look like of a 'direct > virtio->rados link' and 'kernel-mounted rbd' > > > > > > -----Original Message----- > To: ceph-users(a)ceph.io > Subject: *****SPAM***** [ceph-users] Are there 'tuned profiles' for > various ceph scenarios? > > Hi > > Are there any 'official' or even 'works for us' pointers to 'tuned > profiles' for such common uses as > > 'ceph baremetal osd host' > > 'ceph osd + libvirt host' > > 'ceph mon/mgr' > > 'guest vm based on a kernel-mounted rbd' > > 'guest vm based on a direct virtio->rados link' > > I suppose there are a few other common configurations, but you get the > idea. > > If you haven't used or know of 'tuned'-- it's a nice way to collect a > great whole lot of sysctl and other low level configuration options in > one spot. https://tuned-project.org/ > > Thanks > > Harry Coin > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

Here's an example for SCSI disks (the main benefit vs VirtIO is discard/unmap/TRIM support): <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' io='threads' discard='unmap'/> <auth username='libvirt'> <secret type='ceph' uuid='3fc5de60-358e-e010-93c9-bacfc9636edf'/> </auth> <source protocol='rbd' name='yourpool/yourrbd'> <host name='yourmon1' port='6789'/> <host name='yourmon2' port='6789'/> <host name='yourmon3' port='6789'/> </source> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> You also need a VirtIO-SCSI controller to use these, which will look something like: <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> Cheers, Simon. On 01/07/2020 20:52, Harry G. Coin wrote: > [Resent to correct title] > Marc: > > Here's a template that works here.  You'll need to do some steps to > create the 'secret' and make the block devs and so on: > > <disk type="network" device="disk"> >   <driver name="qemu" type="raw" cache="writeback"/> >   <auth username="libvirt"> >     <secret type="ceph" uuid="3fc5de60-358e-e010-93c9-bacfc9636edf"/> >   </auth> >   <source protocol="rbd" name="yournamespace/yourblockdevname" index="1"> >     <host name="yourmon1" port="6789"/> >     <host name="yourmon2" port="6789"/> >     <host name="yourmon3" port="6789"/> >     <host name="yourmon4" port="6789"/> >   </source> >   <target dev="vda" bus="virtio"/> >   <alias name="virtio-disk0"/> >   <address type="pci" domain="0x0000" bus="0x00" slot="0x07" > function="0x0"/> > </disk> > > Glad I could contribute something.   Sure would appreciate leads for the > suggested sysctls/etc either apart or as tuned profiles. > > Harry > > On 7/1/20 2:44 PM, Marc Roos wrote: >> >> Just curious, how does the libvirt xml part look like of a 'direct >> virtio->rados link' and 'kernel-mounted rbd' >> >> >> >> >> >> -----Original Message----- >> To: ceph-users(a)ceph.io >> Subject: *****SPAM***** [ceph-users] Are there 'tuned profiles' for >> various ceph scenarios? >> >> Hi >> >> Are there any 'official' or even 'works for us' pointers to 'tuned >> profiles' for such common uses as >> >> 'ceph baremetal osd host' >> >> 'ceph osd + libvirt host' >> >> 'ceph mon/mgr' >> >> 'guest vm based on a kernel-mounted rbd' >> >> 'guest vm based on a direct virtio->rados link' >> >> I suppose there are a few other common configurations, but you get the >> idea. >> >> If you haven't used or know of 'tuned'-- it's a nice way to collect a >> great whole lot of sysctl and other low level configuration options in >> one spot. https://tuned-project.org/ >> >> Thanks >> >> Harry Coin >> >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io >> >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

Hi, We use KRBD. Our ceph nautilus storage cluster is used for our proxmox 6.x cloud VM disks. This is a bit different as we are not experiencing any errors during boots/reboots we had no unplanned outage, etc., we just safely upgraded from luminous to nautilus and we only experience this when we try to create openvz ploops. I have checked the list but I cant seem to find a solution here. So, we are running nautilus 14.2.9. You can deploy on a VM: https://download.openvz.org/virtuozzo/releases/openvz-7.0.14-136/x86_64/iso… And just try to create an openvz template cache which will try to add a ploop. vzpkg install template centos-7-x86_64 vzpkg create cache centos-7-x86_64 Or check complete usage: https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_command_line_reference… Wrote that here as I thought maybe someone could try reproducing this. Best Regards, ------------------------------------------------------------------------ Daniel Stan Senior System Administrator | NAV Communications (RO) Office: +40 (21) 655-55-55 | E-Mail: daniel(a)nav.ro Site: www.nav.ro <https://www.nav.ro> | Client: https://client.ro On 01/07/2020 15:23, Jason Dillaman wrote: