Cephfs over internet
Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode. Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0] Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong? Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs. From my understanding from the documenation this would be the way to set this up with ceph exposed to internet. Is there something that we are missing or something that would make the setup more secure? Many thanks in advance Marcus
Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode.
Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure
Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong?
Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs.
From my understanding from the documenation this would be the way to set this up with ceph exposed to internet.
Is there something that we are missing or something that would make the setup more secure?
What about a tunnel, and have a local ip range route through it? I am not sure what happens if someone is brute forcing your monitors.
Yeah, you really want to do this over a vpn. Performance is going to be average at best. It would probably be faster to re-export it as NFS/SMB and push that across the internet. On Mon, May 20, 2024 at 11:37 PM Marc <Marc@f1-outsourcing.eu> wrote:
Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode.
Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure
Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong?
Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs.
From my understanding from the documenation this would be the way to set this up with ceph exposed to internet.
Is there something that we are missing or something that would make the setup more secure?
What about a tunnel, and have a local ip range route through it? I am not sure what happens if someone is brute forcing your monitors.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your answers! I read somewhere that a vpn would really have an impact on performance, so it was not recommended, and I found v2 protocol. But vpn feels like the solution and you have to accept the lower speed. Thanks again! On tis, maj 21 2024 at 17:07:48 +1000, Malcolm Haak <insanemal@gmail.com> wrote:
Yeah, you really want to do this over a vpn.
Performance is going to be average at best. It would probably be faster to re-export it as NFS/SMB and push that across the internet.
On Mon, May 20, 2024 at 11:37 PM Marc <Marc@f1-outsourcing.eu <mailto:Marc@f1-outsourcing.eu>> wrote:
Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode.
Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure
Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong?
Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs.
From my understanding from the documenation this would be the way to set this up with ceph exposed to internet.
Is there something that we are missing or something that would make the setup more secure?
What about a tunnel, and have a local ip range route through it? I am not sure what happens if someone is brute forcing your monitors.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
We did a proof of concept moving some compute into "the cloud" and exported our cephfs shares using wireguard as the tunnel. The performance impact on our storage was completely latency and bandwidth dependent with no noticeable impact from the tunnel itself. -paul -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology ________________________________________ From: Marcus <marcus@marcux.org> Sent: Tuesday, May 21, 2024 7:39 AM To: ceph-users Subject: [ceph-users] Re: Cephfs over internet Thanks for your answers! I read somewhere that a vpn would really have an impact on performance, so it was not recommended, and I found v2 protocol. But vpn feels like the solution and you have to accept the lower speed. Thanks again! On tis, maj 21 2024 at 17:07:48 +1000, Malcolm Haak <insanemal@gmail.com> wrote:
Yeah, you really want to do this over a vpn.
Performance is going to be average at best. It would probably be faster to re-export it as NFS/SMB and push that across the internet.
On Mon, May 20, 2024 at 11:37 PM Marc <Marc@f1-outsourcing.eu <mailto:Marc@f1-outsourcing.eu>> wrote:
Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode.
Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure
Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong?
Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs.
From my understanding from the documenation this would be the way to set this up with ceph exposed to internet.
Is there something that we are missing or something that would make the setup more secure?
What about a tunnel, and have a local ip range route through it? I am not sure what happens if someone is brute forcing your monitors.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 5/21/24 13:39, Marcus wrote:
Thanks for your answers! I read somewhere that a vpn would really have an impact on performance, so it was not recommended, and I found v2 protocol. But vpn feels like the solution and you have to accept the lower speed.
Also keep in mind that clients have to be able to access all nodes of the cluster. You are not only exposing the ceph mons, but all ceph services. Using VPN/tunnel is the minimal acceptable solution. Using a NFS/SMB gateway (and expose this gateway via tunnel/VPN + kerberos) should IMHO be a better solution security wise. Regards, Burkhard
Hello, You will want to do this over WireGuard tech from experience, IOPS will be brutal, like 200 IOPS. Wireguard has a few benefits but notably: - Higher rate of transfer per CPU load. - State of the the art protocols. As opposed to some of the more legacy systems. - Extremely fast re-connections, re-keys on IKE can be brutal. Further fine tuning is possible too but i'm, not at that level of knowledge. For flat files of several gigs you're going to get a decent experience, but if you try to run VMs off this... you're in for a bad time. Regards, Adam On 5/21/24 01:07, Malcolm Haak wrote:
Yeah, you really want to do this over a vpn.
Performance is going to be average at best. It would probably be faster to re-export it as NFS/SMB and push that across the internet.
On Mon, May 20, 2024 at 11:37 PM Marc<Marc@f1-outsourcing.eu> wrote:
Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode.
Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0]
Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure
Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong?
Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs.
From my understanding from the documenation this would be the way to set this up with ceph exposed to internet.
Is there something that we are missing or something that would make the setup more secure?
What about a tunnel, and have a local ip range route through it? I am not sure what happens if someone is brute forcing your monitors.
_______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
Hello, To save on power in my home lab can I have a single node CEPH cluster sit idle and powered off for 3 months at a time then boot only to refresh backups? Or will this cause issues I'm unaware of? I'm aware deep-scrubbing will not happen, it would be done when in the boot-up period ever 3 months. What do you think? Regards, Adam
If you have a single node arguably ZFS would be a better choice.
On May 21, 2024, at 14:53, adam.ther <adam.ther@aol.com> wrote:
Hello,
To save on power in my home lab can I have a single node CEPH cluster sit idle and powered off for 3 months at a time then boot only to refresh backups? Or will this cause issues I'm unaware of? I'm aware deep-scrubbing will not happen, it would be done when in the boot-up period ever 3 months.
What do you think?
Regards,
Adam _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think it is his lab so maybe it is a test setup for production. I don't think it matters to much with scrubbing, it is not like it is related to how long you were offline. It will scrub just as much being 1 month offline as being 6 months offline.
If you have a single node arguably ZFS would be a better choice.
On May 21, 2024, at 14:53, adam.ther <adam.ther@aol.com> wrote:
Hello,
To save on power in my home lab can I have a single node CEPH cluster sit idle and powered off for 3 months at a time then boot only to refresh backups? Or will this cause issues I'm unaware of? I'm aware deep- scrubbing will not happen, it would be done when in the boot-up period ever 3 months.
What do you think?
Regards,
Adam _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think it is his lab so maybe it is a test setup for production.
Home production?
I don't think it matters to much with scrubbing, it is not like it is related to how long you were offline. It will scrub just as much being 1 month offline as being 6 months offline.
If you have a single node arguably ZFS would be a better choice.
On May 21, 2024, at 14:53, adam.ther <adam.ther@aol.com> wrote:
Hello,
To save on power in my home lab can I have a single node CEPH cluster sit idle and powered off for 3 months at a time then boot only to refresh backups? Or will this cause issues I'm unaware of? I'm aware deep- scrubbing will not happen, it would be done when in the boot-up period ever 3 months.
What do you think?
Regards,
Adam _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, It's all non-corperate data, I'm just trying to cut back on wattage (removes around 450W of the 2.4KW) by powering down backup servers that house 208TB while not being backed up or restoring. ZFS sounds interesting however does it play nice with a mix of drive sizes? That's primarily why I use Ceph, it's okay (if not ideal) with 4x 22TB, 8x 10TB, 10x 4TB. So that said, would Ceph have any known issues with long power downs aside from it nagging about the scrubbing schedule? Mark i see you said it wouldn't matter but does Ceph not use a date based scheduler? Thanks, Adam On 5/21/24 13:29, Marc wrote:
I think it is his lab so maybe it is a test setup for production. Home production? A home setup to test on, before he applies changes to his production
Saluti 🍷 ;)
It's all non-corperate data, I'm just trying to cut back on wattage (removes around 450W of the 2.4KW) by powering down backup servers that
450W for one server seems quite hefty. Under full load? You can also check your cpu power states and frequency that cuts also some power.
So that said, would Ceph have any known issues with long power downs aside from it nagging about the scrubbing schedule? Mark i see you said it wouldn't matter but does Ceph not use a date based scheduler?
I did not test this, but does not make much sense implementing something that x amount of scrubs need to be done in a specific period. It is an interval 7 days or so, so if it is scrubbed, it is on 0 and your 7 day wait starts.
I would normally vouch for ZFS for this sort of thing, but the mix of drive sizes will be... and inconvenience, at best. You could get creative with the hierarchy (making zraid{2,3} of mirrors of same-sized drives, or something), but it would be far from ideal. I use ZFS for my own home machines however, all the drives are identical. I'm curious about this application of Ceph though, in home-lab use. Performance likely isn't a top concern, just a durable persistent storage target, so this is an interesting use case. On 2024-05-21 17:02, adam.ther wrote:
Hello,
It's all non-corperate data, I'm just trying to cut back on wattage (removes around 450W of the 2.4KW) by powering down backup servers that house 208TB while not being backed up or restoring.
ZFS sounds interesting however does it play nice with a mix of drive sizes? That's primarily why I use Ceph, it's okay (if not ideal) with 4x 22TB, 8x 10TB, 10x 4TB.
So that said, would Ceph have any known issues with long power downs aside from it nagging about the scrubbing schedule? Mark i see you said it wouldn't matter but does Ceph not use a date based scheduler?
Thanks,
Adam
On 5/21/24 13:29, Marc wrote:
I think it is his lab so maybe it is a test setup for production. Home production? A home setup to test on, before he applies changes to his production
Saluti 🍷 ;)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks guys, I think ill just risk it cause it's just for backup, then write something up later as a follow up on what happens in-case others want to do similar. I agree it not typical, im a bit of an odd-duck datahorder. Regards, Adam On 5/21/24 14:21, Matt Vandermeulen wrote:
I would normally vouch for ZFS for this sort of thing, but the mix of drive sizes will be... and inconvenience, at best. You could get creative with the hierarchy (making zraid{2,3} of mirrors of same-sized drives, or something), but it would be far from ideal. I use ZFS for my own home machines however, all the drives are identical.
I'm curious about this application of Ceph though, in home-lab use. Performance likely isn't a top concern, just a durable persistent storage target, so this is an interesting use case.
On 2024-05-21 17:02, adam.ther wrote:
Hello,
It's all non-corperate data, I'm just trying to cut back on wattage (removes around 450W of the 2.4KW) by powering down backup servers that house 208TB while not being backed up or restoring.
ZFS sounds interesting however does it play nice with a mix of drive sizes? That's primarily why I use Ceph, it's okay (if not ideal) with 4x 22TB, 8x 10TB, 10x 4TB.
So that said, would Ceph have any known issues with long power downs aside from it nagging about the scrubbing schedule? Mark i see you said it wouldn't matter but does Ceph not use a date based scheduler?
Thanks,
Adam
On 5/21/24 13:29, Marc wrote:
I think it is his lab so maybe it is a test setup for production. Home production? A home setup to test on, before he applies changes to his production
Saluti 🍷 ;)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It’s usually no problem to shut down a cluster. Set at least the noout flag, the other flags like norebalance, nobackfill etc won’t hurt either. Then shut down the servers. I do that all the time with test clusters (they do have data, just not important at all), and I’ve never had data loss after powering them back on. When all OSDs are up, unset the flags and let it scrub. Usually, the (deep-)scrubbing will start almost immediately. Zitat von "adam.ther" <adam.ther@aol.com>:
Thanks guys,
I think ill just risk it cause it's just for backup, then write something up later as a follow up on what happens in-case others want to do similar. I agree it not typical, im a bit of an odd-duck datahorder.
Regards,
Adam
On 5/21/24 14:21, Matt Vandermeulen wrote:
I would normally vouch for ZFS for this sort of thing, but the mix of drive sizes will be... and inconvenience, at best. You could get creative with the hierarchy (making zraid{2,3} of mirrors of same-sized drives, or something), but it would be far from ideal. I use ZFS for my own home machines however, all the drives are identical.
I'm curious about this application of Ceph though, in home-lab use. Performance likely isn't a top concern, just a durable persistent storage target, so this is an interesting use case.
On 2024-05-21 17:02, adam.ther wrote:
Hello,
It's all non-corperate data, I'm just trying to cut back on wattage (removes around 450W of the 2.4KW) by powering down backup servers that house 208TB while not being backed up or restoring.
ZFS sounds interesting however does it play nice with a mix of drive sizes? That's primarily why I use Ceph, it's okay (if not ideal) with 4x 22TB, 8x 10TB, 10x 4TB.
So that said, would Ceph have any known issues with long power downs aside from it nagging about the scrubbing schedule? Mark i see you said it wouldn't matter but does Ceph not use a date based scheduler?
Thanks,
Adam
On 5/21/24 13:29, Marc wrote:
I think it is his lab so maybe it is a test setup for production. Home production? A home setup to test on, before he applies changes to his production
Saluti 🍷 ;)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, May 21, 2024 at 08:54:26PM +0000, Eugen Block wrote:
It’s usually no problem to shut down a cluster. Set at least the noout flag, the other flags like norebalance, nobackfill etc won’t hurt either. Then shut down the servers. I do that all the time with test clusters (they do have data, just not important at all), and I’ve never had data loss after powering them back on. When all OSDs are up, unset the flags and let it scrub. Usually, the (deep-)scrubbing will start almost immediately.
One surprise I have with the Ubuntu test cluster (non-containerized, Ubuntu packages) that I regularly shut down is that the signals from log rotation (I assume) to Ceph daemons interfere with Ceph startup. When rebooted after some days, it is the same on all nodes: no Ceph daemon is running. Workaround: another reboot Matthias
Zitat von "adam.ther" <adam.ther@aol.com>:
Thanks guys,
I think ill just risk it cause it's just for backup, then write something up later as a follow up on what happens in-case others want to do similar. I agree it not typical, im a bit of an odd-duck datahorder.
Regards,
Adam
On 5/21/24 14:21, Matt Vandermeulen wrote:
I would normally vouch for ZFS for this sort of thing, but the mix of drive sizes will be... and inconvenience, at best. You could get creative with the hierarchy (making zraid{2,3} of mirrors of same-sized drives, or something), but it would be far from ideal. I use ZFS for my own home machines however, all the drives are identical.
I'm curious about this application of Ceph though, in home-lab use. Performance likely isn't a top concern, just a durable persistent storage target, so this is an interesting use case.
On 2024-05-21 17:02, adam.ther wrote:
Hello,
It's all non-corperate data, I'm just trying to cut back on wattage (removes around 450W of the 2.4KW) by powering down backup servers that house 208TB while not being backed up or restoring.
ZFS sounds interesting however does it play nice with a mix of drive sizes? That's primarily why I use Ceph, it's okay (if not ideal) with 4x 22TB, 8x 10TB, 10x 4TB.
So that said, would Ceph have any known issues with long power downs aside from it nagging about the scrubbing schedule? Mark i see you said it wouldn't matter but does Ceph not use a date based scheduler?
Thanks,
Adam
On 5/21/24 13:29, Marc wrote:
> I think it is his lab so maybe it is a test setup for production. Home production? A home setup to test on, before he applies changes to his production
Saluti 🍷 ;)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have already installed multiple one node ceph cluster with cephfs for non-productive workloads in the last few years. Had no major issue, e.g. once a broken HDD. The question is what kind of EC or replication you will use. Also only powered off the node in a clean and healthy state ;-) What would interest me is, pause the cluster with "ceph osd pause" and send the nodes to hibernate. Joachim Am Mi., 22. Mai 2024 um 07:55 Uhr schrieb Matthias Ferdinand < mf+ml.ceph@mfedv.net>:
On Tue, May 21, 2024 at 08:54:26PM +0000, Eugen Block wrote:
It’s usually no problem to shut down a cluster. Set at least the noout flag, the other flags like norebalance, nobackfill etc won’t hurt either. Then shut down the servers. I do that all the time with test clusters (they do have data, just not important at all), and I’ve never had data loss after powering them back on. When all OSDs are up, unset the flags and let it scrub. Usually, the (deep-)scrubbing will start almost immediately.
One surprise I have with the Ubuntu test cluster (non-containerized, Ubuntu packages) that I regularly shut down is that the signals from log rotation (I assume) to Ceph daemons interfere with Ceph startup. When rebooted after some days, it is the same on all nodes: no Ceph daemon is running. Workaround: another reboot
Matthias
Zitat von "adam.ther" <adam.ther@aol.com>:
Thanks guys,
I think ill just risk it cause it's just for backup, then write something up later as a follow up on what happens in-case others want
do similar. I agree it not typical, im a bit of an odd-duck datahorder.
Regards,
Adam
On 5/21/24 14:21, Matt Vandermeulen wrote:
I would normally vouch for ZFS for this sort of thing, but the mix of drive sizes will be... and inconvenience, at best. You could get creative with the hierarchy (making zraid{2,3} of mirrors of same-sized drives, or something), but it would be far from ideal. I use ZFS for my own home machines however, all the drives are identical.
I'm curious about this application of Ceph though, in home-lab use. Performance likely isn't a top concern, just a durable persistent storage target, so this is an interesting use case.
On 2024-05-21 17:02, adam.ther wrote:
Hello,
It's all non-corperate data, I'm just trying to cut back on wattage (removes around 450W of the 2.4KW) by powering down backup servers that house 208TB while not being backed up or restoring.
ZFS sounds interesting however does it play nice with a mix of drive sizes? That's primarily why I use Ceph, it's okay (if not ideal) with 4x 22TB, 8x 10TB, 10x 4TB.
So that said, would Ceph have any known issues with long power downs aside from it nagging about the scrubbing schedule? Mark i see you said it wouldn't matter but does Ceph not use a date based scheduler?
Thanks,
Adam
On 5/21/24 13:29, Marc wrote:
> > I think it is his lab so maybe it is a test setup for
> Home production? A home setup to test on, before he applies changes to his
to production. production
Saluti 🍷 ;)
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (11)
-
adam.ther
-
Anthony D'Atri
-
Burkhard Linke
-
Eugen Block
-
Joachim Kraftmayer
-
Malcolm Haak
-
Marc
-
Marcus
-
Matt Vandermeulen
-
Matthias Ferdinand
-
Paul Mezzanini