- ceph-users - lists.ceph.io

Dealing with non existing crush-root= after reclassify on ec pools
by Dan 18 May '20

18 May '20

I have reclassified a CRUSH map, using the crushtool to a class based ruleset. I still have an ec pool with an older ec profile with a new non existing crush-root=hdd I already switched the pool’s ruleset over to a newer rule with a newer ec-profile with a correct crush-root But pool ls detail still shows: pool 9 'data' erasure profile jerasure-3-1 size 4 min_size 3 ….. Jerasure-3-1 being the old profile with non existing crush-root So what do I do now? Switching over the pool ruleset does not change the ec-profile, can I switch the ec-profile over? What can I expect having a pool with a ec-profile with a non existing crush-root key? Please advise.

1 0

how to restart daemons on 15.2 on Debian 10
by Ml Ml 18 May '20

18 May '20

Hello List, how do you restart daemons (mgr, mon, osd) on 15.2.1? It used to be something like: systemctl stop ceph-osd@10 Or: systemctl start ceph-mon@ceph03 however, those command do nothing on my setup. Is this because i use cephadm and that docker stuff? The Logs also seem to be missing. /var/log/ceph/5436dd5d-83d4-4dc8-a93b-60ab5db145df is pretty empty. I feel like i am missing a lot of documentation here? Can anyone point me to my missing parts? Thanks a lot. Cheers, Michael

3 6

Re: Cephadm and rados gateways
by brendan＠tobolaski.com 18 May '20

18 May '20

I'm experiencing a similar thing with Cephadm when attempting to deploy rgw. Following the instructions at https://ceph.readthedocs.io/en/latest/cephadm/install/#deploy-rgws results in the container failing to start up immediately. Digging through the logs, it is because it can't bind to port 80. There is no configuration on my machines that would prevent it from doing so nor is there an existing service bound to 0.0.0.0:80. Checking within the container, I see that the ceph binary does not have cap_net_bind_service which would be required to bind to port 80 from a non-root user. In addition, I see that rgw_frontends is not editable and is set to port 80. This seems like a bug in cephadm / the containers it uses. Either the containers should allow ceph to bind to port 80 or rgw_frontends should remain at the default of 7480 to allow the frontends to be deployed. Is there a way for me to work around this?

2 1

Luminous to Nautilus mon upgrade oddity - failed to decode mgrstat state; luminous dev version? buffer::end_of_buffer
by Thomas Byrne - UKRI STFC 18 May '20

18 May '20

Hi all, We're upgrading a cluster from luminous to nautilus. The monitors and managers are running a non-release version of luminous (12.2.12-642-g5ff3e8e) and we're upgrading them to 14.2.9. We've upgraded one monitor and it's happily in quorum as a peon. However, when a ceph status hits the nautilus mon it has trouble talking to the manager apparently, and it returns a status output with no pg stats and garbage usage numbers. From the mon log: 2020-05-13 10:41:43.121 7fa1e6fdf700 0 mon.ceph-mon5@4(peon) e25 handle_command mon_command({"prefix": "status"} v 0) v1 2020-05-13 10:41:43.121 7fa1e6fdf700 0 log_channel(audit) log [DBG] : from='client.? v1:130.246.x.x:0/3261311028' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2020-05-13 10:41:43.443 7fa1e6fdf700 -1 mon.ceph-mon5(a)4(peon).mgrstat failed to decode mgrstat state; luminous dev version? buffer::end_of_buffer 2020-05-13 10:41:44.397 7fa1e6fdf700 1 mon.ceph-mon5@4(peon) e25 dropping unexpected mon_health( e 0 r 0 ) v1 Is this expected for a luminous to nautilus upgrade or could this be due to the odd luminous version we are running, or something else entirely? Cheers, Tom

2 2

Re: Ceph as a Fileserver for 3D Content Production
by Janne Johansson 18 May '20

18 May '20

Den sön 17 maj 2020 kl 14:33 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: > outs, and you are more likely to shoot yourself in the foot. At least > ask first. Eg this bcache, I am not 100% sure what it is, but if it is > sitting between the osd process and the disk, it could be getting nasty > with a reset/power outage, when ceph thinks data is written to disk, > while it is not. > Bcache is a neat way in linux to stack devices, working basically like the ceph layered pools, but on a per-host basis of course. The main advantage, and why some ceph admins use if instead of putting WAL/DB on the fast device, is that a single bcache device can be placed in front of several other devices, and the kernel will decide which writes are going to use the write cache, so in effect you are sharing the full size of your fast device speed on all OSDs, instead of having to split your NVME into pieces or preset a journal size. If you at any point expect to add more drives to a box, this becomes even more interesting, since knowing beforehand how many preallocations to leave room for is very hard and leaves out potential gains until you get it right. -- May the most significant bit of your life be positive.

2 1

RGW issue with containerized ceph
by Szabo, Istvan (Agoda) 18 May '20

18 May '20

Hi Gents, We are having strange problem with demo ceph container. Any write operation we do from the RGW fail with a NameResolutionFailure Read operation work fine. If we use the ceph UI we can create objects fine though. This is the env varibles we are using to configure it: This is the full compose: version: '3.3' services: ceph: ports: - 8080:8080 # S3 - 5000:5000 # Web UI - 8003:8003 environment: - NETWORK_AUTO_DETECT=4 - CEPH_DAEMON=demo - RGW_NAME=ceph - CEPH_DEMO_UID=owncloud - CEPH_DEMO_ACCESS_KEY=G1EZ5R4K6IJ7XUQKMAED - CEPH_DEMO_SECRET_KEY=cNmUrqpBKjCMzcfqG8fg4Qk07Xkoyau52OmvnSsz - DEBUG=verbose image: ceph/daemon Is there anybody has some idea what is going on? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

OSDs taking too much memory, for pglog
by Harald Staub 18 May '20

18 May '20

Several OSDs of one of our clusters are down currently because RAM usage has increased during the last days. Now it is more than we can handle on some systems. Frequently OSDs get killed by the OOM killer. Looking at "ceph daemon osd.$OSD_ID dump_mempools", it shows that nearly all (about 8.5 GB) is taken by osd_pglog, e.g. "osd_pglog": { "items": 461859, "bytes": 8445595868 }, We tried to reduce it, with "osd memory target" and even with "bluestore cache autotune = false" (together with "bluestore cache size hdd"), but there was no effect at all. I remember the pglog_hardlimit parameter, but that is already set by default with Nautilus I read. I.e. this is on Nautilus, 14.2.8. Is there a way to limit this pglog memory? Cheers Harry

4 6

Need help on cache tier monitoring
by icy chan 18 May '20

18 May '20

Hi, I had configured a cache tier with below parameters: cache_target_dirty_ratio: 0.1 cache_target_dirty_high_ratio: 0.7 cache_target_full_ratio: 0.9 The cache tier did improved the performance much. And I am targeted to keep the cache tier with only 10% of dirty data. Remaining data ( 80% ) are clean and ready for evict. I don't want the cache tier enter high flushing state which impacting the performance. Are there any methods to keep track the dirty ratio of target cache tier? Which I can plan for adding more OSDs if the dirty ratio keep floating around 50~70% Regs, Icy

1 1

Ceph-mgr wont start, cant find rook module
by Jeff Welling 18 May '20

18 May '20

Hello my ceph-using comrades! I've been using ceph for awhile at home but wanted to update to the latest, Octopus. I got it installed on a single node, added a second node and some OSDs, and have been migrating from my original Jewel cluster. When I installed Octopus, I wiped the systems and installed Debian Buster, added the ceph apt repos instead of using the packages in Debian, and installed manually using ceph-vol to create Bluestore OSDs, using the ceph docs as my guide. Now though, one of the two Octopus nodes (the one running ceph-mgr and ceph-mon) are crashing weekly. I haven't been able to look into the cause of the crashes in detail yet as these are hobbyist systems and work has been exceptionally busy lately, but now after the most recent crash, I'm unable to start ceph-mgr and the syslog has ceph-mgr messages complaining of not being able to find the 'rook' module. This is rather confusing because though I'm aware of rook, to my knowledge I've never used it on my systems, and there's no mention of it in my config. I tried applying pending upgrades but that hasn't changed the behavior. I normally wouldn't dare ask for help this early in my adventure but I find myself in a bit of a pinch. By any chance have you hit this before, or know what may be causing it? Ceph is awesome. Keep up the good work, stay safe, and Thank You Kindly in advance! My ceph version root@zim:~# ceph --version ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) This is my ceph.config [global] fsid = 495d7f30-CCCC-BBBB-AAAA-ddf6ffe063d0 mon initial members = zim.internal.justdev.ca mon host = 192.168.0.11 public network = 192.168.0.0/24 cluster network = 192.168.42.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 333 osd pool default pgp num = 333 osd crush chooseleaf type = 1 rbd_default_features = 7 These are the syslogs that show up when trying to restart ceph-mgr May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.065-0700 7fcdccaa2f40 -1 mgr[py] Module not found: 'rook' May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.065-0700 7fcdccaa2f40 -1 mgr[py] Traceback (most recent call last): May 4 01:35:13 zim ceph-mgr[21602]: File "/usr/share/ceph/mgr/rook/__init__.py", line 2, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from .module import RookOrchestrator May 4 01:35:13 zim ceph-mgr[21602]: File "/usr/share/ceph/mgr/rook/module.py", line 16, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from kubernetes import client, config May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/__init__.py", line 22, in <module> May 4 01:35:13 zim ceph-mgr[21602]: import kubernetes.stream May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/stream/__init__.py", line 15, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from .stream import stream May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/stream/stream.py", line 13, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import ws_client May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/kubernetes/stream/ws_client.py", line 19, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from websocket import WebSocket, ABNF, enableTrace May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/websocket/__init__.py", line 22, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from ._abnf import * May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/websocket/_abnf.py", line 34, in <module> May 4 01:35:13 zim ceph-mgr[21602]: import numpy May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import core May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/core/__init__.py", line 40, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import multiarray May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/core/multiarray.py", line 12, in <module> May 4 01:35:13 zim ceph-mgr[21602]: from . import overrides May 4 01:35:13 zim ceph-mgr[21602]: File "/lib/python3/dist-packages/numpy/core/overrides.py", line 65, in <module> May 4 01:35:13 zim ceph-mgr[21602]: """) May 4 01:35:13 zim ceph-mgr[21602]: RuntimeError: _get_implementing_args method already has a docstring May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.069-0700 7fcdccaa2f40 -1 mgr[py] Class not found in module 'rook' May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.069-0700 7fcdccaa2f40 -1 mgr[py] Error loading module 'rook': (2) No such file or directory May 4 01:35:13 zim ceph-mgr[21602]: 2020-05-04T01:35:13.673-0700 7fcdccaa2f40 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook

2 2

Ceph as a Fileserver for 3D Content Production
by Moritz Wilhelm 17 May '20

17 May '20

Hi Martin, The Samsung Evo is just my system drive on my test host. This is my server setup 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME __________________________________ EventArt GmbH Kirchenstraße 13, 3243 St. Leonhard am Forst mobile: +43(0)6641951196 mail: office(a)eventart.at<mailto:office@eventart.at> UID: ATU 64417269 http://www.eventart.at II http://www.clamlive.at Von: Martin Verges <martin.verges(a)croit.io> Gesendet: Sonntag, 17. Mai 2020 18:28 An: Moritz Wilhelm <moritz(a)eventart.at> Cc: Marc Roos <M.Roos(a)f1-outsourcing.eu>; ceph-users <ceph-users(a)ceph.io> Betreff: Re: [ceph-users] Re: Ceph as a Fileserver for 3D Content Production Hello Moritz, drop the EVO disk and use a SSD that is working with Ceph. For example just use PM883 / PM983 from the same vendor and you will have a huge performance increase. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.verges(a)croit.io<mailto:martin.verges@croit.io> Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx Am So., 17. Mai 2020 um 15:12 Uhr schrieb Moritz Wilhelm <moritz(a)eventart.at<mailto:moritz@eventart.at>>: Hi Marc, thank you very much for your feedback, actually that is what I am looking for (Design advices and feedback). I also wanted to get in touch with the community because right now I am on my own with this project with no experience at all. But I also wanted to get into it first and learn the basics and setup a system before bothering other people with it and to be able to even hold a conversation about it. So we are a 3D Content Production Company with around 10-12 Workstations and 6 Render stations. We work with software like Autodesk Maya or Adobe After effects. All our Projects are stored on the NAS and we work directly with those files on the NAS. This is our current System: Synology RS4017xs+ Intel Xeon D-1541 2,1 Ghz, 8 Cores 32GB Ram 16x 8TB WD Red Drives 2 TB SSD Cache 2 x 10 Gb SFP+ Nics About the Bcache I just did tests with it as I found the idea very interesting and also performance was better. If this project is ever getting into real-life production I will stick to the most common setups for sure. Here are some Results of Diskmark on one Computer. (I Had only 5 OSDs per Node up and running at this test because I am still waiting for the new Seagate drives) For Sequential Reads and Writes I sometimes maxed out the 10Gbe host connection so the data is not very useful but the other numbers are quite interesting: For Comparison: Hosts Local NVME Drive (Samsung 970 EVO Pro 1TB): Read: Write: Sequential 1MiB (Q= 8, T= 1): 3507.700 MB/s [ 3345.2 IOPS] < 2389.61 us> Sequential 1MiB (Q= 8, T= 1): 2548.063 MB/s [ 2430.0 IOPS] < 3284.79 us> Sequential 1MiB (Q= 1, T= 1): 2368.308 MB/s [ 2258.6 IOPS] < 442.07 us> Sequential 1MiB (Q= 1, T= 1): 2456.471 MB/s [ 2342.7 IOPS] < 426.10 us> Random 4KiB (Q= 32, T=16): 1555.565 MB/s [ 379776.6 IOPS] < 1312.45 us> Random 4KiB (Q= 32, T=16): 16.580 MB/s [ 4047.9 IOPS] <124267.88 us> Random 4KiB (Q= 1, T= 1): 51.666 MB/s [ 12613.8 IOPS] < 78.82 us> Random 4KiB (Q= 1, T= 1): 108.983 MB/s [ 26607.2 IOPS] < 37.13 us> Current Synology NAS (SMB): Read: Write: Sequential 1MiB (Q= 8, T= 1): 1045.594 MB/s [ 997.2 IOPS] < 7990.33 us> Sequential 1MiB (Q= 8, T= 1): 1101.007 MB/s [ 1050.0 IOPS] < 7588.25 us> Sequential 1MiB (Q= 1, T= 1): 953.709 MB/s [ 909.5 IOPS] < 1098.57 us> Sequential 1MiB (Q= 1, T= 1): 847.847 MB/s [ 808.6 IOPS] < 1235.26 us> Random 4KiB (Q= 32, T=16): 4.198 MB/s [ 1024.9 IOPS] <380158.65 us> Random 4KiB (Q= 32, T=16): 188.827 MB/s [ 46100.3 IOPS] < 11076.14 us> Random 4KiB (Q= 1, T= 1): 2.486 MB/s [ 606.9 IOPS] < 1637.08 us> Random 4KiB (Q= 1, T= 1): 7.177 MB/s [ 1752.2 IOPS] < 570.16 us> Ceph With WAL/DB on NVME total of 5 old SATA HDD OSDs (SMB): Read: Write: Sequential 1MiB (Q= 8, T= 1): 534.050 MB/s [ 509.3 IOPS] < 15628.63 us> Sequential 1MiB (Q= 8, T= 1): 198.420 MB/s [ 189.2 IOPS] < 42020.67 us> Sequential 1MiB (Q= 1, T= 1): 340.580 MB/s [ 324.8 IOPS] < 2921.17 us> Sequential 1MiB (Q= 1, T= 1): 184.329 MB/s [ 175.8 IOPS] < 5603.99 us> Random 4KiB (Q= 32, T=16): 3.172 MB/s [ 774.4 IOPS] <398622.73 us> Random 4KiB (Q= 32, T=16): 8.639 MB/s [ 2109.1 IOPS] <222699.43 us> Random 4KiB (Q= 1, T= 1): 1.907 MB/s [ 465.6 IOPS] < 2139.08 us> Random 4KiB (Q= 1, T= 1): 7.294 MB/s [ 1780.8 IOPS] < 560.91 us> Ceph With Bcache total of 5 old SATA HDD OSDs (SMB): Sequential 1MiB (Q= 8, T= 1): 967.386 MB/s [ 922.6 IOPS] < 8660.59 us> Sequential 1MiB (Q= 8, T= 1): 882.882 MB/s [ 842.0 IOPS] < 9444.23 us> Sequential 1MiB (Q= 1, T= 1): 418.157 MB/s [ 398.8 IOPS] < 2448.00 us> Sequential 1MiB (Q= 1, T= 1): 793.417 MB/s [ 756.7 IOPS] < 1320.37 us> Random 4KiB (Q= 32, T=16): 39.756 MB/s [ 9706.1 IOPS] < 51788.10 us> Random 4KiB (Q= 32, T=16): 128.303 MB/s [ 31324.0 IOPS] < 16298.94 us> Random 4KiB (Q= 1, T= 1): 29.527 MB/s [ 7208.7 IOPS] < 138.36 us> Random 4KiB (Q= 1, T= 1): 32.334 MB/s [ 7894.0 IOPS] < 126.25 us> I will do an Update as soon as I have all the 18 OSDs running. I could also need some advice on how to do proper testing. __________________________________ EventArt GmbH Kirchenstraße 13, 3243 St. Leonhard am Forst mobile: +43(0)6641951196 mail: office(a)eventart.at<mailto:office@eventart.at> UID: ATU 64417269 http://www.eventart.at II http://www.clamlive.at -----Ursprüngliche Nachricht----- Von: Marc Roos <M.Roos(a)f1-outsourcing.eu<mailto:M.Roos@f1-outsourcing.eu>> Gesendet: Sonntag, 17. Mai 2020 14:33 An: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>>; Moritz Wilhelm <moritz(a)eventart.at<mailto:moritz@eventart.at>> Betreff: RE: [ceph-users] Ceph as a Fileserver for 3D Content Production I was just reading your post, and started wondering why you posted it. I do not see clear question, and you also do not share test results (from your nas vs cephfs smb). So maybe you like some attention in this covid social distancing time? ;) Anyway, I have been 'testing' with ceph for 2,5 years or so. (I am telling you so, you know how to value my info) If you start playing with ceph keep it simple and stick to what is recommended. Do not start looking for unconventional ways making ceph faster, because you are not able to do it better than the developers. You do not know the ins and outs, and you are more likely to shoot yourself in the foot. At least ask first. Eg this bcache, I am not 100% sure what it is, but if it is sitting between the osd process and the disk, it could be getting nasty with a reset/power outage, when ceph thinks data is written to disk, while it is not. I would start by identifying what your minimum performance requirements are, maybe post them, and ask if someone has realized them with a ceph setup. -----Original Message----- Sent: 17 May 2020 13:42 To: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Ceph as a Fileserver for 3D Content Production Hi, my Name is Moritz and I am working for a 3D production company. Because of the corona virus I have too much time left and also to much unused hardware. That is why I started playing around with Ceph as a fileserver for us. Here I want to share my experience for all those who are interested. To start of here is my actual running test system. I am interested in the thoughts of the community and also on more suggestions on what to try out with my available Hardware. I don’t know how to test it right now because I am a newbie to ceph and our production file server is a super user-friendly but high performance Synology NAS 😉. All I have done so far was running Crystal disk benchmark on 1 Windows machine on the SMB Share. 3 Nodes: (original those where render workstations that are not in use right now) Each Node is MON MGM OSD Mainboard: ASRock TRX40 Creator CPU: AMD Ryzen Threadripper 3960X, 24 Cores, 3.8Ghz RAM: 2 x Samsung 32 GB 2 x 8 DDR4 2666 MHz 288-pin DIMM, Unregistered, ECC (64 GB Total) NIC Public: OnBoard Aquantia 107, 10Gbit NIC Ceph: Intel XXV710-DA2, 2x SFP28, 25Gbit System Drive: 2x Samsung SSD 860 PRO 256GB, SATA, ZFS Raid 1 System: Proxmox VE 6.2, Debian Buster, Ceph Nautilus HBA: Broadcom SAS 9305-16i OSDs: 6x Seagate Exos, 16TB, 7.200 rpm, 12Gb SAS Cache: 1x Micron 9300 MAX 3.2TB U.2 NVME I Played around with setting it up as a WAL/DB Device. Right now I have configured the Micron NVME as a BCache Infront of the 6 Seagate Drives in writeback mode. Because in this configuration BCache takes care of translating random writes to sequential ones for the HDDs I turned the Ceph WAL LOG off. I think Bcache gives more options to tune the System for my use case instead of just putting WAL/DB on the NVME. And also I can easily add cache drives or remove them without touching osds. I set up SMB Shares with the vfs_ceph module. I still have to add CTDB to distribute Samba to all nodes. My Next steps are to keep playing around in tuning the system and testing stability and performance. After that I want to put the Ceph cluster infront of our production NAS. Because our data is not super critical I thought of setting the replicas to 2 and running Rsync overnight to our NAS. That way I can switch to the old NAS at any time and wouldn’t loose more than 1 Day of work which is acceptable for us. This is how I could compare the two solutions side by side with real-life workload. I know that ceph might not be the best solution right now but if I am able to get at least similar performance to our Synology HDD NAS out of it, it would give a super scalable Solution in size and performance to grow with our needs. And who knows what performance improvements we get with ceph in the next 3 years. I am happy to hear your thoughts and ideas. And please I know this might be kind of a crazy setup but I have fun with it and I learned a lot the last few weeks. If my experiment fails I will go back to my original plan: Put FreeNas on two of the Nodes with overnight replication and put the third Node back to his render-friends. 😃 By the way I also have a spare Dell Server: 2x Xeon E5-2630 v3 2,40GHz, 128G Ram. I just don’t have an idea on how to utilize it. Maybe as extra OSD Node or as a separate Samba Server to get the SMB traffic away from the Public Ceph Network. Moritz Wilhelm _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0