Ceph newbee questions
Hi all, I am all new with ceph and I come from gluster. We have had our eyes on ceph for several years and as the gluster project seems to slow down we now think it is time to start look into ceph. I have manually configured a ceph cluster with ceph fs on debian bookworm. What is the difference from installing with cephadm compared to manuall install, any benefits that you miss with manual install? There are also another couple of things that I can not figure out reading the documentation. Most of our files are small and from my understanding replication is then recomended, right? The plan is to set ceph up like this: 1 x "admin node" 2 x "storage nodes" The admin node will run mon, mgr and mds. The storage nodes will run mon, mgr, mds and 8x osd (8 disks). This works well to setup but I can not get my head around is how things are replicated over nodes and disks. In ceth.conf I set the folowing: osd pool default size = 2 osd pool default min size = 1 So the idea is that we always have 2 copies of the data. I do not seem to be able to figure out the replication when things starts to fail. If the admin node goes down, one of the data nodes will run the mon, mgr and mds. This will slow things down but will be fine until we have a new admin node in place again. (or if there is something I am missing here?) If just one data node goes down we will still not loose any data and that is fine until we have a new server. But what if one data node goes down and one disk of the other data node breaks, will I loose data then? Or how many disks can I loose before I loose data? This is what I can not get my head around, how to think when disaster strikes, how much hardware can I loose before I loose data? Or have I got it all wrong? Is it a bad idea with just 2 fileservers is more servers required? The second thing I have a problem with is snapshots. I manage to create snapshot in root with command: ceph fs subvolume snapshot create <vol_name> / <snap_name> But it fails if I try to create a shapshot in any other directory then in the root. Second of all if I try to create a snapshot from the client with: mkdir /mnt-ceph/.snap/my_snapshot I get the same error in all directories: Permission dened. I have not found any sollution to this, am I missing something here as well? Any config missing? Many thanks for your support!! Best regrads Marcus
I have manually configured a ceph cluster with ceph fs on debian bookworm.
Bookworm support is very, very recent I think.
What is the difference from installing with cephadm compared to manuall install, any benefits that you miss with manual install?
A manual install is dramatically more work and much easier to get wrong. There's also Rook if you skate k8s.
There are also another couple of things that I can not figure out reading the documentation.
Most of our files are small and from my understanding replication is then recomended, right?
How small is "small"? https://docs.google.com/spreadsheets/d/1rpGfScgG-GLoIGMJWDixEkqs-On9w8nAUToP... If your files are super small, like say <256KB you may consume measurably more underlying storage space than you expect. CephFS isn't my strong suit, but my understanding is that it's designed for reasonably large files. As with RGW, if you store zillions of 1KB files you may not have the ideal experience.
The plan is to set ceph up like this: 1 x "admin node"
MDS AIUI is single-threaded and so will benefit from a high-frequency CPU more than a high-core-count CPU.
2 x "storage nodes"
You can do that for a PoC, but that's a bad idea for any production workload. You'd want at least three nodes with OSDs to use the default RF=3 replication. You can do RF=2, but at the peril of your mortal data.
This works well to setup but I can not get my head around is how things are replicated over nodes and disks. In ceth.conf I set the folowing: osd pool default size = 2 osd pool default min size = 1 So the idea is that we always have 2 copies of the data.
Those are only defaults if you don't specify them when creating a pool. I suggest always specifying the replication parameters explicitly when creating a pool. min_size = 1 is a trap for any data you care about.
I do not seem to be able to figure out the replication when things starts to fail. If the admin node goes down, one of the data nodes will run the mon, mgr and mds. This will slow things down but will be fine until we have a new admin node in place again. (or if there is something I am missing here?)
If you have 3 mons, that's mostly true. The MDS situation is more nuanced.
If just one data node goes down we will still not loose any data and that is fine until we have a new server.
... unless one of the drives in the surviving node fails.
But what if one data node goes down and one disk of the other data node breaks, will I loose data then?
It most likely will be at least unavailable until you get the first node back up with all OSDs. This is one reason why RF=2 is okay for a sandbox but a bad idea for any data you care about. There are legit situations where one doesn't care so much about losing data, but they are infrequent.
Or how many disks can I loose before I loose data? This is what I can not get my head around, how to think when disaster strikes, how much hardware can I loose before I loose data? Or have I got it all wrong? Is it a bad idea with just 2 fileservers is more servers required?
Ceph is a scale-out solution, not meant for a very small number of servers. For replication, you really want at least 3 nodes with OSDs and size=3,min_size=2. More nodes is better. If you need a smaller-scale solution, DRBD or ZFS might be better choices.
The second thing I have a problem with is snapshots. I manage to create snapshot in root with command: ceph fs subvolume snapshot create <vol_name> / <snap_name> But it fails if I try to create a shapshot in any other directory then in the root. Second of all if I try to create a snapshot from the client with: mkdir /mnt-ceph/.snap/my_snapshot I get the same error in all directories: Permission dened. I have not found any sollution to this, am I missing something here as well? Any config missing?
Many thanks for your support!!
Best regrads Marcus
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Disclaimer: I'm fairly new to Ceph, but I've read a bunch of threads on the min_size=1 issue as that was perplexing me when I started, as one replica is generally considered fine in many other applications. However, there really are some unique concerns to Ceph beyond just the number of disks you can lose... On Fri, Dec 22, 2023 at 3:09 PM Anthony D'Atri <aad@dreamsnake.net> wrote:
2 x "storage nodes"
You can do that for a PoC, but that's a bad idea for any production workload. You'd want at least three nodes with OSDs to use the default RF=3 replication. You can do RF=2, but at the peril of your mortal data.
I'm not sure I agree - I think size=2, min_size=2 is no worse than RAID1 for data security. Maybe some consider RAID1 inappropriate, and if so then size=2 is the same, but I think many are quite comfortable with it. The issue is that if you lose a disk your PGs become inactive - the data is perfectly safe, but you have downtime. Of course that is probably not what you're expecting, as that isn't what happens with RAID1. Read on...
If just one data node goes down we will still not loose any data and that is fine until we have a new server.
... unless one of the drives in the surviving node fails.
That isn't even the main risk as I understand it. Of course a double failure is going to be a problem with size=2, or traditional RAID1, and I think anybody choosing this configuration accepts this risk. As I understand it, the reason min_size=1 is a trap has nothing to do with double failures per se. The issue is that Ceph OSDs are somewhat prone to flapping during recovery (OOM, etc). So even if the disk is fine, an OSD can go down for a short time. If you have size=2, min=1 configured, then when this happens the PG will become degraded and will continue operating on the other OSD, and the flapping OSD becomes stale. Then when it comes back up it recovers. The problem is that if the other OSD has a permanent failure (disk crash/etc) while the first OSD is flapping, now you have no good OSDs, because when the flapping OSD comes back up it is stale, and its PGs have no peer. I suspect there are ways to re-activate it, though this will result in potential data inconsistency since writes were allowed to the cluster and will then get rolled back. With only two OSDs I'm guessing that would be the main impact (well, depending on journaling behavior/etc), but if you have more OSDs than that then you could have situations where one file is getting rolled back, and some other file isn't, and so on. With min_size=2 you're fairly safe from flapping because there will always be two replicas that have the most recent version of every PG, and so you can still tolerate a permanent failure of one of them. size=2, min=2 doesn't suffer this failure mode, because anytime there is flapping the PG goes inactive and no writes can be made, so when the other OSD comes back up there is nothing to recover. Of course this results in IO blocks and downtime, which is obviously undesirable, but it is likely a more recoverable state than inconsistent writes. Apologies if I've gotten any of that wrong, but my understanding is that it is these sorts of failure modes that cause min_size=1 to be a trap. This isn't the sort of thing that typically happens in a RAID1 config, or at least that admins don't think about. Those implementations are simpler and less prone to flapping, though to its credit I'm guessing ceph would be far better about detecting this sort of thing in the first place.
Ceph is a scale-out solution, not meant for a very small number of servers. For replication, you really want at least 3 nodes with OSDs and size=3,min_size=2. More nodes is better. If you need a smaller-scale solution, DRBD or ZFS might be better choices.
Agree that pretty-much all distributed filesystems suffer performance issues at the very least with a small number of nodes, especially with hard disks. Moosefs is a decent distributed solution on small clusters, but it lacks high availability on the FOSS version. I've found that on small clusters on hard disks it actually performs much better than cephfs, but it certainly won't scale up nearly as well. With only 2-3 hard disks though it still will perform fairly poorly. Of course ZFS will perform best of all, but it lacks any kind of host-level redundancy. -- Rich
participants (3)
-
Anthony D'Atri
-
Marcus
-
Rich Freeman