- ceph-users - lists.ceph.io

Re: Ceph as a Fileserver for 3D Content Production
by Martin Verges 17 May '20

17 May '20

Hello Moritz, drop the EVO disk and use a SSD that is working with Ceph. For example just use PM883 / PM983 from the same vendor and you will have a huge performance increase. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am So., 17. Mai 2020 um 15:12 Uhr schrieb Moritz Wilhelm <moritz(a)eventart.at >: > Hi Marc, > > thank you very much for your feedback, actually that is what I am looking > for (Design advices and feedback). I also wanted to get in touch with the > community because right now I am on my own with this project with no > experience at all. But I also wanted to get into it first and learn the > basics and setup a system before bothering other people with it and to be > able to even hold a conversation about it. > > So we are a 3D Content Production Company with around 10-12 Workstations > and 6 Render stations. We work with software like Autodesk Maya or Adobe > After effects. All our Projects are stored on the NAS and we work directly > with those files on the NAS. > > This is our current System: > > Synology RS4017xs+ > Intel Xeon D-1541 2,1 Ghz, 8 Cores > 32GB Ram > 16x 8TB WD Red Drives > 2 TB SSD Cache > 2 x 10 Gb SFP+ Nics > > > About the Bcache I just did tests with it as I found the idea very > interesting and also performance was better. If this project is ever > getting into real-life production I will stick to the most common setups > for sure. > > Here are some Results of Diskmark on one Computer. (I Had only 5 OSDs per > Node up and running at this test because I am still waiting for the new > Seagate drives) For Sequential Reads and Writes I sometimes maxed out the > 10Gbe host connection so the data is not very useful but the other numbers > are quite interesting: > > For Comparison: > > Hosts Local NVME Drive (Samsung 970 EVO Pro 1TB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 3507.700 MB/s [ 3345.2 IOPS] < > 2389.61 us> Sequential 1MiB (Q= 8, T= 1): 2548.063 MB/s [ > 2430.0 IOPS] < 3284.79 us> > Sequential 1MiB (Q= 1, T= 1): 2368.308 MB/s [ 2258.6 IOPS] < > 442.07 us> Sequential 1MiB (Q= 1, T= 1): 2456.471 MB/s [ > 2342.7 IOPS] < 426.10 us> > Random 4KiB (Q= 32, T=16): 1555.565 MB/s [ 379776.6 IOPS] < > 1312.45 us> Random 4KiB (Q= 32, T=16): 16.580 MB/s [ > 4047.9 IOPS] <124267.88 us> > Random 4KiB (Q= 1, T= 1): 51.666 MB/s [ 12613.8 IOPS] < > 78.82 us> Random 4KiB (Q= 1, T= 1): 108.983 MB/s [ > 26607.2 IOPS] < 37.13 us> > > Current Synology NAS (SMB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 1045.594 MB/s [ 997.2 IOPS] < > 7990.33 us> Sequential 1MiB (Q= 8, T= 1): 1101.007 MB/s [ > 1050.0 IOPS] < 7588.25 us> > Sequential 1MiB (Q= 1, T= 1): 953.709 MB/s [ 909.5 IOPS] < > 1098.57 us> Sequential 1MiB (Q= 1, T= 1): 847.847 MB/s [ > 808.6 IOPS] < 1235.26 us> > Random 4KiB (Q= 32, T=16): 4.198 MB/s [ 1024.9 IOPS] > <380158.65 us> Random 4KiB (Q= 32, T=16): 188.827 MB/s [ > 46100.3 IOPS] < 11076.14 us> > Random 4KiB (Q= 1, T= 1): 2.486 MB/s [ 606.9 IOPS] < > 1637.08 us> Random 4KiB (Q= 1, T= 1): 7.177 MB/s > [ 1752.2 IOPS] < 570.16 us> > > Ceph With WAL/DB on NVME total of 5 old SATA HDD OSDs (SMB): > Read: > Write: > Sequential 1MiB (Q= 8, T= 1): 534.050 MB/s [ 509.3 IOPS] < > 15628.63 us> Sequential 1MiB (Q= 8, T= 1): 198.420 MB/s [ > 189.2 IOPS] < 42020.67 us> > Sequential 1MiB (Q= 1, T= 1): 340.580 MB/s [ 324.8 IOPS] < > 2921.17 us> Sequential 1MiB (Q= 1, T= 1): 184.329 MB/s [ > 175.8 IOPS] < 5603.99 us> > Random 4KiB (Q= 32, T=16): 3.172 MB/s [ 774.4 IOPS] > <398622.73 us> Random 4KiB (Q= 32, T=16): 8.639 MB/s [ > 2109.1 IOPS] <222699.43 us> > Random 4KiB (Q= 1, T= 1): 1.907 MB/s [ 465.6 IOPS] < > 2139.08 us> Random 4KiB (Q= 1, T= 1): 7.294 MB/s > [ 1780.8 IOPS] < 560.91 us> > > Ceph With Bcache total of 5 old SATA HDD OSDs (SMB): > Sequential 1MiB (Q= 8, T= 1): 967.386 MB/s [ 922.6 IOPS] < > 8660.59 us> Sequential 1MiB (Q= 8, T= 1): 882.882 MB/s [ > 842.0 IOPS] < 9444.23 us> > Sequential 1MiB (Q= 1, T= 1): 418.157 MB/s [ 398.8 IOPS] < > 2448.00 us> Sequential 1MiB (Q= 1, T= 1): 793.417 MB/s [ > 756.7 IOPS] < 1320.37 us> > Random 4KiB (Q= 32, T=16): 39.756 MB/s [ 9706.1 IOPS] < > 51788.10 us> Random 4KiB (Q= 32, T=16): 128.303 MB/s [ > 31324.0 IOPS] < 16298.94 us> > Random 4KiB (Q= 1, T= 1): 29.527 MB/s [ 7208.7 IOPS] < > 138.36 us> Random 4KiB (Q= 1, T= 1): 32.334 MB/s [ > 7894.0 IOPS] < 126.25 us> > > I will do an Update as soon as I have all the 18 OSDs running. > > I could also need some advice on how to do proper testing. > > > > > > __________________________________ > > EventArt GmbH > Kirchenstraße 13, > 3243 St. Leonhard am Forst > mobile: +43(0)6641951196 > mail: office(a)eventart.at > UID: ATU 64417269 > > http://www.eventart.at II http://www.clamlive.at > > -----Ursprüngliche Nachricht----- > Von: Marc Roos <M.Roos(a)f1-outsourcing.eu> > Gesendet: Sonntag, 17. Mai 2020 14:33 > An: ceph-users <ceph-users(a)ceph.io>; Moritz Wilhelm <moritz(a)eventart.at> > Betreff: RE: [ceph-users] Ceph as a Fileserver for 3D Content Production > > > > I was just reading your post, and started wondering why you posted it. I > do not see clear question, and you also do not share test results (from > your nas vs cephfs smb). So maybe you like some attention in this covid > social distancing time? ;) > > Anyway, I have been 'testing' with ceph for 2,5 years or so. (I am telling > you so, you know how to value my info) If you start playing with ceph keep > it simple and stick to what is recommended. Do not start looking for > unconventional ways making ceph faster, because you are not able to do it > better than the developers. You do not know the ins and outs, and you are > more likely to shoot yourself in the foot. At least ask first. Eg this > bcache, I am not 100% sure what it is, but if it is sitting between the osd > process and the disk, it could be getting nasty with a reset/power outage, > when ceph thinks data is written to disk, while it is not. > > I would start by identifying what your minimum performance requirements > are, maybe post them, and ask if someone has realized them with a ceph > setup. > > > > > -----Original Message----- > Sent: 17 May 2020 13:42 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Ceph as a Fileserver for 3D Content Production > > Hi, > > my Name is Moritz and I am working for a 3D production company. Because of > the corona virus I have too much time left and also to much unused > hardware. That is why I started playing around with Ceph as a fileserver > for us. Here I want to share my experience for all those who are > interested. To start of here is my actual running test system. I am > interested in the thoughts of the community and also on more suggestions on > what to try out with my available Hardware. I don’t know how to test it > right now because I am a newbie to ceph and our production file server is a > super user-friendly but high performance Synology NAS 😉. > All I have done so far was running Crystal disk benchmark on 1 Windows > machine on the SMB Share. > > 3 Nodes: (original those where render workstations that are not in use > right now) > > Each Node is MON MGM OSD > > Mainboard: ASRock TRX40 Creator > CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz > RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC > (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: > Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO > 256GB, SATA, ZFS Raid 1 > System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus > HBA: Broadcom SAS 9305-16i > > OSDs: > 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS > > Cache: > 1x Micron 9300 MAX 3.2TB U.2 NVME > > I Played around with setting it up as a WAL/DB Device. Right now I have > configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in > writeback mode. > Because in this configuration BCache takes care of translating random > writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I > think Bcache gives more options to tune the System for my use case instead > of just putting WAL/DB on the NVME. And also I can easily add cache drives > or remove them without touching osds. > > I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to > distribute Samba to all nodes. > > My Next steps are to keep playing around in tuning the system and testing > stability and performance. After that I want to put the Ceph cluster > infront of our production NAS. Because our data is not super critical I > thought of setting the replicas to 2 and running Rsync overnight to our > NAS. That way I can switch to the old NAS at any time and wouldn’t loose > more than 1 Day of work which is acceptable for us. > This is how I could compare the two solutions side by side with real-life > workload. > > I know that ceph might not be the best solution right now but if I am able > to get at least similar performance to our Synology HDD NAS out of it, it > would give a super scalable Solution in size and performance to grow with > our needs. And who knows what performance improvements we get with ceph in > the next 3 years. > > I am happy to hear your thoughts and ideas. And please I know this might > be kind of a crazy setup but I have fun with it and I learned a lot the > last few weeks. If my experiment fails I will go back to my original > plan: Put FreeNas on two of the Nodes with overnight replication and put > the third Node back to his render-friends. 😃 > > By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, > 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra > OSD Node or as a separate Samba Server to get the SMB traffic away from the > Public Ceph Network. > > > > > Moritz Wilhelm > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Ceph as a Fileserver for 3D Content Production
by Marc Roos 17 May '20

17 May '20

I was just reading your post, and started wondering why you posted it. I do not see clear question, and you also do not share test results (from your nas vs cephfs smb). So maybe you like some attention in this covid social distancing time? ;) Anyway, I have been 'testing' with ceph for 2,5 years or so. (I am telling you so, you know how to value my info) If you start playing with ceph keep it simple and stick to what is recommended. Do not start looking for unconventional ways making ceph faster, because you are not able to do it better than the developers. You do not know the ins and outs, and you are more likely to shoot yourself in the foot. At least ask first. Eg this bcache, I am not 100% sure what it is, but if it is sitting between the osd process and the disk, it could be getting nasty with a reset/power outage, when ceph thinks data is written to disk, while it is not. I would start by identifying what your minimum performance requirements are, maybe post them, and ask if someone has realized them with a ceph setup. -----Original Message----- Sent: 17 May 2020 13:42 To: ceph-users(a)ceph.io Subject: [ceph-users] Ceph as a Fileserver for 3D Content Production Hi, my Name is Moritz and I am working for a 3D production company. Because of the corona virus I have too much time left and also to much unused hardware. That is why I started playing around with Ceph as a fileserver for us. Here I want to share my experience for all those who are interested. To start of here is my actual running test system. I am interested in the thoughts of the community and also on more suggestions on what to try out with my available Hardware. I don’t know how to test it right now because I am a newbie to ceph and our production file server is a super user-friendly but high performance Synology NAS 😉. All I have done so far was running Crystal disk benchmark on 1 Windows machine on the SMB Share. 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME I Played around with setting it up as a WAL/DB Device. Right now I have configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in writeback mode. Because in this configuration BCache takes care of translating random writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I think Bcache gives more options to tune the System for my use case instead of just putting WAL/DB on the NVME. And also I can easily add cache drives or remove them without touching osds. I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to distribute Samba to all nodes. My Next steps are to keep playing around in tuning the system and testing stability and performance. After that I want to put the Ceph cluster infront of our production NAS. Because our data is not super critical I thought of setting the replicas to 2 and running Rsync overnight to our NAS. That way I can switch to the old NAS at any time and wouldn’t loose more than 1 Day of work which is acceptable for us. This is how I could compare the two solutions side by side with real-life workload. I know that ceph might not be the best solution right now but if I am able to get at least similar performance to our Synology HDD NAS out of it, it would give a super scalable Solution in size and performance to grow with our needs. And who knows what performance improvements we get with ceph in the next 3 years. I am happy to hear your thoughts and ideas. And please I know this might be kind of a crazy setup but I have fun with it and I learned a lot the last few weeks. If my experiment fails I will go back to my original plan: Put FreeNas on two of the Nodes with overnight replication and put the third Node back to his render-friends. 😃 By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra OSD Node or as a separate Samba Server to get the SMB traffic away from the Public Ceph Network. Moritz Wilhelm _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 2

Ceph as a Fileserver for 3D Content Production
by Moritz Wilhelm 17 May '20

17 May '20

Hi, my Name is Moritz and I am working for a 3D production company. Because of the corona virus I have too much time left and also to much unused hardware. That is why I started playing around with Ceph as a fileserver for us. Here I want to share my experience for all those who are interested. To start of here is my actual running test system. I am interested in the thoughts of the community and also on more suggestions on what to try out with my available Hardware. I don’t know how to test it right now because I am a newbie to ceph and our production file server is a super user-friendly but high performance Synology NAS 😉. All I have done so far was running Crystal disk benchmark on 1 Windows machine on the SMB Share. 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME I Played around with setting it up as a WAL/DB Device. Right now I have configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in writeback mode. Because in this configuration BCache takes care of translating random writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I think Bcache gives more options to tune the System for my use case instead of just putting WAL/DB on the NVME. And also I can easily add cache drives or remove them without touching osds. I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to distribute Samba to all nodes. My Next steps are to keep playing around in tuning the system and testing stability and performance. After that I want to put the Ceph cluster infront of our production NAS. Because our data is not super critical I thought of setting the replicas to 2 and running Rsync overnight to our NAS. That way I can switch to the old NAS at any time and wouldn’t loose more than 1 Day of work which is acceptable for us. This is how I could compare the two solutions side by side with real-life workload. I know that ceph might not be the best solution right now but if I am able to get at least similar performance to our Synology HDD NAS out of it, it would give a super scalable Solution in size and performance to grow with our needs. And who knows what performance improvements we get with ceph in the next 3 years. I am happy to hear your thoughts and ideas. And please I know this might be kind of a crazy setup but I have fun with it and I learned a lot the last few weeks. If my experiment fails I will go back to my original plan: Put FreeNas on two of the Nodes with overnight replication and put the third Node back to his render-friends. 😃 By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra OSD Node or as a separate Samba Server to get the SMB traffic away from the Public Ceph Network. Moritz Wilhelm

1 0

Re: Zeroing out rbd image or volume
by ceph＠elchaka.de 17 May '20

17 May '20

Hello Samuel, Why you want to do that? Just remove the rbd - as long your Image is distributet over random OSDs there is no way to recovery a delete Image.. ...2 cents Mehmet Am 12. Mai 2020 12:55:22 MESZ schrieb "huxiaoyu(a)horebdata.cn" <huxiaoyu(a)horebdata.cn>: >Hi， Ceph folks, > >Is there a rbd command, or any other way, to zero out rbd images or >volume? I would like to write all zero data to an rbd image/volume >before remove it. > >Any comments would be appreciated. > >best regards, > >samuel >Horebdata AG >Switzerland > > > > >huxiaoyu(a)horebdata.cn >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

[ceph-users]
by Amit Ghadge 16 May '20

16 May '20

Hi All, I have question regarding Ceph Nautilus upgrade. In our test environment upgrading Luminous to Nautilus 14.2.8, and after enable msgr2, we seen one of the mon node restarted, my question is this normal process of restart mon service and 2nd question is, we using below mon_host format that can be correct or move to v2? mon_host = 10.44.172.181,10.44.172.182,10.44.172.183 Thanks, AmitG

1 0

Cephfs - NFS Ganesha
by Amudhan P 16 May '20

16 May '20

Hi, I am trying to setup NFS ganesh in Ceph Nautilus. In a ubuntu 18.04 system i have installed nfs-ganesha (v2.6) and nfs-ganesha-ceph pkg and followed the steps in the link https://docs.ceph.com/docs/nautilus/cephfs/nfs/ but i am not able to export my cephfs volume there is no error msg in nfs-ganesha, also i doubt whether its loading nfs-ganesha-ceph config file from "/etc/ganesha" folder. From same system i am able to mount thru ceph kernel client without any issue? How do i make this work? regards Amudhan

3 4

Re: EC Plugins Benchmark with Current Intel/AMD CPU
by Lazuardi Nasution 15 May '20

15 May '20

Hi Marc, I read https://blog.dachary.org/2015/05/12/ceph-jerasure-and-isa-plugins-benchmark… and it seem there is significant performance difference between benchmarked plugins. But, it was with old Intel E3-1200v2 series. I have read some others EC benchmark result improvements with current CPU but I cannot find it for Ceph implementation. Best regards, On Fri, May 15, 2020 at 9:03 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > How many % of the latency is even CPU related? > > > > -----Original Message----- > From: Lazuardi Nasution [mailto:mrxlazuardin@gmail.com] > Sent: 15 May 2020 16:00 > To: ceph-users(a)ceph.io > Subject: [ceph-users] EC Plugins Benchmark with Current Intel/AMD CPU > > Hi, > > Is there any EC plugins benchmark with current Intel/AMD CPU? It seem > there are new instructions which may accelerate EC. Let's say we want to > benchmark plugins using Intel 6200 or AMD 7002 series. I hope there is > better result than what have been benchmarked some years ago. > > Best regards, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > >

1 0

EC Plugins Benchmark with Current Intel/AMD CPU
by Lazuardi Nasution 15 May '20

15 May '20

Hi, Is there any EC plugins benchmark with current Intel/AMD CPU? It seem there are new instructions which may accelerate EC. Let's say we want to benchmark plugins using Intel 6200 or AMD 7002 series. I hope there is better result than what have been benchmarked some years ago. Best regards,

2 1

Ceph modules
by Alfredo De Luca 15 May '20

15 May '20

Hi all. Just a quick one. Are there any ansible modules for ceph around? Cheers -- */Alfredo*

2 1

ACL for user in another teant
by Vishwas Bm 15 May '20

15 May '20

> Hi, > > I have two users both belong to different tenant. > > Can I give permission for the user in another tenant to access the bucket > using setacl or setPolicy command ? > I tried the setacl command and setpolicy command, but it was not working ? > It used to say bucket not found, when the grantee tried to access. > > Is this supported ? > > *Thanks & Regards,* > *Vishwas * > >

2 5