On Wed, Jun 2, 2021 at 12:31 PM Willem Jan Withagen <wjw(a)digiware.nl> wrote: Currently, the proposal is to remove it in the R release (one release after deprecation), which should be in March 2023. Would that give you enough time to migrate? - Neha

On 6/26/21 12:53 AM, Stuart Longland wrote: FWIW, you can lower both the osd_memory_target and tweak a couple of other settings that will lower bluestore memory usage.  A 2GB target is about the lowest you can reasonably set it to (and you'll likely hurt performance due to cache misses), but saying you need a host with 8+GB of RAM is probably a little excessive. There's also a good chance that filestore memory usage isn't as consistently low as you think it is.  Yes you can avoid the in-memory caches that bluestore has since filestore relies more heavily on page cache, but things like osdmap, pglog, and various other buffers are still going to use memory in filestore just like bluestore.  You might find yourself working fine 99% of the time and then going OOM during recovery or something if you try to deploy filestore on a low memory SBC. Having said all of that, you can get 16GB of ECC DDR4 RAM in the US new for around $70-100USD.  A quick search on google makes it look like you'll pay about twice that new in AU, but there's plenty of stuff on the used market for ~$60-100AUD (like $50-70USD).  I don't think that's super unreasonable and frankly would be far more reliable than running on SBCs with non-ECC memory.  I would love to see SBCs become more prolific, but memory has always been a big constraint (especially before the 8GB devices came out), and not only for Ceph. Mark

On Sat, 26 Jun 2021 10:06:10 -0700 Anthony D'Atri <anthony.datri(a)gmail.com> wrote: Yeah, I wouldn't want to go below 4GB RAM. Originally when I deployed my 3-node cluster, I was getting comparable performance to Microsoft Azure's cheaper offerings. (Not a glowing endorcement of their cloud I might add, but it was quite acceptable.) Never used HIPPI. A 48-port gigabit managed switch is reasonably accessible to the home gamer, both in terms of availability and cost. Second-hand 10GbE switches can be found for reasonable prices, but a new one is pricey! Too expensive for my liking. … because Ceph needs one interface for the "public" network and one for the "private" network? Plus, 802.3AD helps. Truth be told, I'd like to ditch the BMCs, but most BIOSes have a fixation of needing a monitor and keyboard to configure them. CoreBoot has the right idea, but isn't widely available on kit accessible to the home experimenter. Well, my use case is a small-scale cluster in a SOHO-type environment. It started out as a project at my workplace to investigate how to set up a small private cloud arrangement, then I replicated the set-up at home to better explore the options with a view of applying what I had learned to the cluster at my workplace. So, not production in the sense a business runs on it, but my mail server and numerous other workloads do run from this cluster. A nice feature over a RAID system is that I can bring one node down for maintenance, and still be "online", albeit with degraded performance. In this case, 3 of my nodes are running two OSDs: Samsung SSD 860 2TB (Bluestore) and WDC WD20SPZX-00U (Filestore). Built on these boards: https://www.supermicro.com/products/motherboard/atom/X10/A1SAi-2750F.cfm and mounted up in a DIN-rail mounted case. (Presently with OS running off a USB-3.0 external drive.) I added two more to give me breathing room when re-deploying nodes (in particular, going from Filestore on BTRFS to Bluestore, then back to Filestore on XFS), these just have one WDC WD20SPZX-00U each (also Filestore). These were built on Intel NUCs because I needed them in a hurry and I had some DDR4 SO-DIMMs that I bought by mistake. So that's 5 WDC WD20SPZX-00U OSDs and 3 Samsung SSD 860 OSDs. I'm looking to move out of the DIN-rail cases as it looks like I'm out-growing them, so maybe in the future I might replace these with 3.5" drives, but right now this is what I have. Filestore may not set the world on fire, and may be worse off in bigger deployments, but it works in the smaller ones really well from what I've seen. -- Stuart Longland (aka Redhatter, VK4MSL) I haven't lost my mind... ...it's backed up on a tape somewhere.

FWIW, I have been running several production clusters with osd_memory_target = 3GB without issues (with Nautilus / Bluestore, 4TB to 12TB hard drives and from 300 to 1400 OSDs per cluster). I have an external caching layer for hot data though, so that cache misses on Ceph are the common case and not an issue. Eric

If you want to go cheap and somewhat questionable, there are some asrock mainboards with a soldered in atom cpu, that support up to 32gb memory (officially only 8, but the controller does more) and have 2 sata directly + a free 16x pcie port, Those boards are usually less than 90€, not as cheap as a raspberry, but you can do way larger nodes due a faster cpu and more memory. You could even add additional network ports via usb3.... But I would not use something like this for anything more serious than a proof of concept system/ home nas. Greetings On 6/28/21 2:05 AM, Stuart Longland wrote: > On Sat, 26 Jun 2021 08:01:46 -0500 > Mark Nelson <mnelson(a)redhat.com> wrote: > >> FWIW, you can lower both the osd_memory_target and tweak a couple of >> other settings that will lower bluestore memory usage.  A 2GB target is >> about the lowest you can reasonably set it to (and you'll likely hurt >> performance due to cache misses), but saying you need a host with 8+GB >> of RAM is probably a little excessive. There's also a good chance that >> filestore memory usage isn't as consistently low as you think it is. >> Yes you can avoid the in-memory caches that bluestore has since >> filestore relies more heavily on page cache, but things like osdmap, >> pglog, and various other buffers are still going to use memory in >> filestore just like bluestore.  You might find yourself working fine 99% >> of the time and then going OOM during recovery or something if you try >> to deploy filestore on a low memory SBC. > To be honest, the smallest of my nodes has 8GB RAM presently, but I'd > like to scale out, and most of my cost-effective options for scale-out > are 4GB or less. Raspberry Pi4 is the only I've seen available to mere > mortals like myself that exceeds this limit, and even then it's far from > an ideal system. > > PC Engines APU3s look good as nodes, as they have SATA > on-board, multiple Ethernet interfaces that can be bonded for quick > networking, and they use CoreBoot managed over a serial port, but > needing 8GB is a killer. > > I presently use filestore on HDD-based OSDs because I've found it gives > me the best performance. The SSD-based OSDs are running Bluestore, > since they seem to be able to keep up better. Never tried FileStore on > less than 8GB, so you could well be right, but my experience with BlueStore > on these OSDs was abysmal. > >> Having said all of that, you can get 16GB of ECC DDR4 RAM in the US new >> for around $70-100USD.  A quick search on google makes it look like >> you'll pay about twice that new in AU, but there's plenty of stuff on >> the used market for ~$60-100AUD (like $50-70USD).  I don't think that's >> super unreasonable and frankly would be far more reliable than running >> on SBCs with non-ECC memory.  I would love to see SBCs become more >> prolific, but memory has always been a big constraint (especially before >> the 8GB devices came out), and not only for Ceph. > Yep, but remember when you buy a SBC, the RAM comes > soldered-to-the-board in most cases. If you want removable RAM, you're > looking at a small-form-factor server board of some kind like the > Supermicro A1SAi boards that have been my storage nodes since 2016. > > Regards,