OK what you will read below might sound insane but I am obliged to ask. There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere. I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know. While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list. OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment. Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope? Am I misunderstanding something about IPFS that merits taking a second look at it? Is there any other way of getting scalable reliable "decentralized" internet-wide storage? I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work? -- Linas -- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.
Am 4/5/25 um 05:39 schrieb Linas Vepstas:
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster?
This will not work as the latency is too high. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
On 4/4/25 11:39 PM, Linas Vepstas wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
Similar to climate research data back in 2017... It was all accessible via FTP or HTTP though. A Climate Mirror initiative was created and a distributed copy worldwide was made eventually. Essentially, a list of URLs was provided and some helper scripts to slurp multiple copies of data repositories. https://climatemirror.org/ https://github.com/climate-mirror -- Šarūnas Burdulis Dartmouth Mathematics math.dartmouth.edu/~sarunas · https://useplaintext.email ·
Thanks Šarūnai and all who responded. I guess general discussion will need to go off-list. But first: To summarize, the situation seems to be this: * As a general rule, principle investigators (PI) always have a copy of their "master dataset", which thus is "safe" as long as they don't lose control over it. * Certain data sets are popular and are commonly shared. * NCBI publishes data sets, with the goal of making access easy, transparent, fast, documented, and shoulders the burden of network costs, sysadmin, server maintenance, etc. and it is this "free, easy, managed-for-you" infrastructure that is at risk. * Unlike climate data, some of the NIH data is covered by HIPAA (e.g. cancer datasets) because it contains personal identifying information. I have no clue how this is dealt with. Encryption? Passwords? Restricted access? Who makes the decision about who is allowed, and who is not allowed to work with, access, copy or mirror the data? WTF? I'm clueless here. What are the technical problems to be solved? As long as PI's have a copy of a master dataset, the technical issues are: -- how to find it? -- what does it contain? -- is there enough network bandwidth? -- can it be copied in full? -- if it can be, where's the mirrors / backups? -- If the PI's lab is shut down, who pays for the storage and network connectivity for the backups? -- How to protect against loss of backup copies? -- How to gain access to backup copies? The above issues sit at the "library science" level: yes, technology can help, but it's also social and organizational. So it's not really about "how can we build a utopian decentralized data store" in some abstract way that shards data across multiple nodes (which is what IPFS seemed to want to be). Instead, its four-fold: * How is the catalog of available data maintained? * How is the safety of backup copies ensured? * How do we cache data, improve latency, improve bandwidth? * How are the administrative burdens shared? (sysadmin, cost of servers, bandwidth) This is way far outside of the idea of "let's just harness a bunch of disks together on the internet", but it is the actual problem being faced. -- Linas On Mon, Apr 7, 2025 at 8:07 AM Šarūnas Burdulis <sarunas@math.dartmouth.edu> wrote:
On 4/4/25 11:39 PM, Linas Vepstas wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
Similar to climate research data back in 2017... It was all accessible via FTP or HTTP though. A Climate Mirror initiative was created and a distributed copy worldwide was made eventually. Essentially, a list of URLs was provided and some helper scripts to slurp multiple copies of data repositories.
https://climatemirror.org/ https://github.com/climate-mirror
-- Šarūnas Burdulis Dartmouth Mathematics math.dartmouth.edu/~sarunas
· https://useplaintext.email · _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.
Sounds like a discussion for a discord server. Or BlueSky or something that's very definitely NOT what used to be known as twitter. My viewpoint is a little different. I really didn't consider HIPAA stuff, although since technically that is info that shouldn't be accessible to anyone but authorized staff at NIH - and there's the rub, if the very persons/offices involved are purged. At that point, what we'd really be doing is simply hiding it until a saner regime comes along and wants it back. But it's not just NIH that's being tossed down the Memory Hole. NASA, NOAA, and other agencies are also being "cleansed". We should properly be safeguarding ALL of that. Reminds me of Isaac Asimov's Foundation - an agency to preserve human knowledge over the dark ages. Also, the idea of having fixed homes for complete documents I feel is limiting. I'm minded of how the folding@home project distributed work to random volunteers. And again, how ceph can break an object into PGs and splatter them to replicas on multiple servers. It's less important for a given document server to be 100% online as it is to have the ability for nodes to check in and out and maintain a gestalt. As for the management of all this, I'd say that the top-level domain of my theoretical namespace would be a select committee in charge of the master servers. sub-domains would be administered by grant from the top level and have their own administrators. And so forth until you have librarian administrators. Existing examples can be seen in some of the larger git archives, such as for Linux. The Wikipedia can also provide examples of how to administer tamper-resistant information. So, in short, I'm proposing a sort of world-wide web of documents. Something that can live in the background of ordinary user computers, perhaps. But most importantly, reliable, accessible and secure. Tim On 4/7/25 15:33, Linas Vepstas wrote:
Thanks Šarūnai and all who responded.
I guess general discussion will need to go off-list. But first:
To summarize, the situation seems to be this: * As a general rule, principle investigators (PI) always have a copy of their "master dataset", which thus is "safe" as long as they don't lose control over it. * Certain data sets are popular and are commonly shared. * NCBI publishes data sets, with the goal of making access easy, transparent, fast, documented, and shoulders the burden of network costs, sysadmin, server maintenance, etc. and it is this "free, easy, managed-for-you" infrastructure that is at risk. * Unlike climate data, some of the NIH data is covered by HIPAA (e.g. cancer datasets) because it contains personal identifying information. I have no clue how this is dealt with. Encryption? Passwords? Restricted access? Who makes the decision about who is allowed, and who is not allowed to work with, access, copy or mirror the data? WTF? I'm clueless here.
What are the technical problems to be solved? As long as PI's have a copy of a master dataset, the technical issues are: -- how to find it? -- what does it contain? -- is there enough network bandwidth? -- can it be copied in full? -- if it can be, where's the mirrors / backups? -- If the PI's lab is shut down, who pays for the storage and network connectivity for the backups? -- How to protect against loss of backup copies? -- How to gain access to backup copies?
The above issues sit at the "library science" level: yes, technology can help, but it's also social and organizational. So it's not really about "how can we build a utopian decentralized data store" in some abstract way that shards data across multiple nodes (which is what IPFS seemed to want to be). Instead, its four-fold:
* How is the catalog of available data maintained? * How is the safety of backup copies ensured? * How do we cache data, improve latency, improve bandwidth? * How are the administrative burdens shared? (sysadmin, cost of servers, bandwidth)
This is way far outside of the idea of "let's just harness a bunch of disks together on the internet", but it is the actual problem being faced.
-- Linas
On Mon, Apr 7, 2025 at 8:07 AM Šarūnas Burdulis <sarunas@math.dartmouth.edu> wrote:
On 4/4/25 11:39 PM, Linas Vepstas wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere. Similar to climate research data back in 2017... It was all accessible via FTP or HTTP though. A Climate Mirror initiative was created and a distributed copy worldwide was made eventually. Essentially, a list of URLs was provided and some helper scripts to slurp multiple copies of data repositories.
https://climatemirror.org/ https://github.com/climate-mirror
-- Šarūnas Burdulis Dartmouth Mathematics math.dartmouth.edu/~sarunas
· https://useplaintext.email · _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tim, Agree w/ all you say. My two cents an old-timer. These ideas have been around for decades. And there have been many plans, attempts, screeds, projects (I can rattle off a random list, but so can search engines and wikipedia). All with good intentions, heart in the right place. All got mired or stuck in one way or another. What does it take to gain critical mass and move forward? The traditional road to success is: -- Build a prototype that works. -- Make sure it solves an actual problem that actual people really have. -- Make it easy to understand. -- Accept patches rapidly. -- Lots of luck, and getting slashdotted. Maybe one of the many existing projects could be adapted, re-formed, re-aimed. Or maybe they're all dead in the water because they failed one of more of the above five bullets. -- Linas On Mon, Apr 7, 2025 at 3:11 PM Tim Holloway <timh@mousetech.com> wrote:
Sounds like a discussion for a discord server. Or BlueSky or something that's very definitely NOT what used to be known as twitter.
My viewpoint is a little different. I really didn't consider HIPAA stuff, although since technically that is info that shouldn't be accessible to anyone but authorized staff at NIH - and there's the rub, if the very persons/offices involved are purged. At that point, what we'd really be doing is simply hiding it until a saner regime comes along and wants it back.
But it's not just NIH that's being tossed down the Memory Hole. NASA, NOAA, and other agencies are also being "cleansed". We should properly be safeguarding ALL of that. Reminds me of Isaac Asimov's Foundation - an agency to preserve human knowledge over the dark ages.
Also, the idea of having fixed homes for complete documents I feel is limiting. I'm minded of how the folding@home project distributed work to random volunteers. And again, how ceph can break an object into PGs and splatter them to replicas on multiple servers. It's less important for a given document server to be 100% online as it is to have the ability for nodes to check in and out and maintain a gestalt.
As for the management of all this, I'd say that the top-level domain of my theoretical namespace would be a select committee in charge of the master servers. sub-domains would be administered by grant from the top level and have their own administrators. And so forth until you have librarian administrators. Existing examples can be seen in some of the larger git archives, such as for Linux. The Wikipedia can also provide examples of how to administer tamper-resistant information.
So, in short, I'm proposing a sort of world-wide web of documents. Something that can live in the background of ordinary user computers, perhaps. But most importantly, reliable, accessible and secure.
Tim
On 4/7/25 15:33, Linas Vepstas wrote:
Thanks Šarūnai and all who responded.
I guess general discussion will need to go off-list. But first:
To summarize, the situation seems to be this: * As a general rule, principle investigators (PI) always have a copy of their "master dataset", which thus is "safe" as long as they don't lose control over it. * Certain data sets are popular and are commonly shared. * NCBI publishes data sets, with the goal of making access easy, transparent, fast, documented, and shoulders the burden of network costs, sysadmin, server maintenance, etc. and it is this "free, easy, managed-for-you" infrastructure that is at risk. * Unlike climate data, some of the NIH data is covered by HIPAA (e.g. cancer datasets) because it contains personal identifying information. I have no clue how this is dealt with. Encryption? Passwords? Restricted access? Who makes the decision about who is allowed, and who is not allowed to work with, access, copy or mirror the data? WTF? I'm clueless here.
What are the technical problems to be solved? As long as PI's have a copy of a master dataset, the technical issues are: -- how to find it? -- what does it contain? -- is there enough network bandwidth? -- can it be copied in full? -- if it can be, where's the mirrors / backups? -- If the PI's lab is shut down, who pays for the storage and network connectivity for the backups? -- How to protect against loss of backup copies? -- How to gain access to backup copies?
The above issues sit at the "library science" level: yes, technology can help, but it's also social and organizational. So it's not really about "how can we build a utopian decentralized data store" in some abstract way that shards data across multiple nodes (which is what IPFS seemed to want to be). Instead, its four-fold:
* How is the catalog of available data maintained? * How is the safety of backup copies ensured? * How do we cache data, improve latency, improve bandwidth? * How are the administrative burdens shared? (sysadmin, cost of servers, bandwidth)
This is way far outside of the idea of "let's just harness a bunch of disks together on the internet", but it is the actual problem being faced.
-- Linas
On Mon, Apr 7, 2025 at 8:07 AM Šarūnas Burdulis <sarunas@math.dartmouth.edu> wrote:
On 4/4/25 11:39 PM, Linas Vepstas wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere. Similar to climate research data back in 2017... It was all accessible via FTP or HTTP though. A Climate Mirror initiative was created and a distributed copy worldwide was made eventually. Essentially, a list of URLs was provided and some helper scripts to slurp multiple copies of data repositories.
https://climatemirror.org/ https://github.com/climate-mirror
-- Šarūnas Burdulis Dartmouth Mathematics math.dartmouth.edu/~sarunas
· https://useplaintext.email · _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.
Maybe one of the many existing projects could be adapted, re-formed, re-aimed. Or maybe they're all dead in the water because they failed one of more of the above five bullets.
Often the unstated sixth bullet: — Has a supportable architecture The ongoing viability of a system should not rely on someone constantly having to monitor and manage contracts with thousands of high-touch low-savvy micro providers.
MooseFS is the way to go here. I have it working on android SD cards and of course normal Linux servers over the internet and over Yggdrasil-network. One of my in-progress anarchy projects is a global hard drive for all of humanity’s knowledge. I would LOVE to get involved with this preservation project technically in a volunteer capacity. I can build a cutting edge resilient distributed storage system for cheaper than anything currently on the market. Please reach out or pass along my email. Alex On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yeah, Ceph in its current form doesn't seem like a good fit. I think that what we need to support the world's knowledge in the face of enstupidification is some sort of distributed holographic datastore. so, like Ceph's PG replication, a torrent-like ability to pull from multiple unreliable sources, a good indexing mechanism and, protections against tampering. Probably with a touch of git as well. I'm sure there's more, but those are items that immediately occur to me. Tim On 4/7/25 09:10, Alex Buie wrote:
MooseFS is the way to go here.
I have it working on android SD cards and of course normal Linux servers over the internet and over Yggdrasil-network.
One of my in-progress anarchy projects is a global hard drive for all of humanity’s knowledge.
I would LOVE to get involved with this preservation project technically in a volunteer capacity. I can build a cutting edge resilient distributed storage system for cheaper than anything currently on the market.
Please reach out or pass along my email.
Alex
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Additional features. * No "master server". No Single Point of Failure. * Resource location. A small number of master servers kept in sync like DNS with tiers of secondary resources. I think blockchains also have a similar setup? * Resource identification. A scheme like LDAP. For example: cn=library,catalog=dewey,filing=504.3,... cn=library,country=us,catalog=libraryofcongress,... country=us,agency=nih,department=... cn=upc,isbn=... A document/document set should have a canonical name, but allow alternate names for ease of location, such as author searches, general topics and the like. I considered OIDs as an alternative, but LDAP names are more human-friendly and easier to add sub-domains to without petitioning a master registrar. Also there's a better option for adding attributes to the entry description. On 4/7/25 09:39, Tim Holloway wrote:
Yeah, Ceph in its current form doesn't seem like a good fit.
I think that what we need to support the world's knowledge in the face of enstupidification is some sort of distributed holographic datastore. so, like Ceph's PG replication, a torrent-like ability to pull from multiple unreliable sources, a good indexing mechanism and, protections against tampering. Probably with a touch of git as well.
I'm sure there's more, but those are items that immediately occur to me.
Tim
On 4/7/25 09:10, Alex Buie wrote:
MooseFS is the way to go here.
I have it working on android SD cards and of course normal Linux servers over the internet and over Yggdrasil-network.
One of my in-progress anarchy projects is a global hard drive for all of humanity’s knowledge.
I would LOVE to get involved with this preservation project technically in a volunteer capacity. I can build a cutting edge resilient distributed storage system for cheaper than anything currently on the market.
Please reach out or pass along my email.
Alex
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Super cool idea - I too wanted to refer to blockchain methods to avoid data being tampered. Ceph would need a completely different distribution coded for such storage, however we could say that the fundamentals are already in place? Best, Laimis J.
On 7 Apr 2025, at 18:23, Tim Holloway <timh@mousetech.com> wrote:
Additional features.
* No "master server". No Single Point of Failure.
* Resource location. A small number of master servers kept in sync like DNS with tiers of secondary resources. I think blockchains also have a similar setup?
* Resource identification. A scheme like LDAP. For example:
cn=library,catalog=dewey,filing=504.3,...
cn=library,country=us,catalog=libraryofcongress,...
country=us,agency=nih,department=...
cn=upc,isbn=...
A document/document set should have a canonical name, but allow alternate names for ease of location, such as author searches, general topics and the like.
I considered OIDs as an alternative, but LDAP names are more human-friendly and easier to add sub-domains to without petitioning a master registrar. Also there's a better option for adding attributes to the entry description.
On 4/7/25 09:39, Tim Holloway wrote:
Yeah, Ceph in its current form doesn't seem like a good fit.
I think that what we need to support the world's knowledge in the face of enstupidification is some sort of distributed holographic datastore. so, like Ceph's PG replication, a torrent-like ability to pull from multiple unreliable sources, a good indexing mechanism and, protections against tampering. Probably with a touch of git as well.
I'm sure there's more, but those are items that immediately occur to me.
Tim
On 4/7/25 09:10, Alex Buie wrote:
MooseFS is the way to go here.
I have it working on android SD cards and of course normal Linux servers over the internet and over Yggdrasil-network.
One of my in-progress anarchy projects is a global hard drive for all of humanity’s knowledge.
I would LOVE to get involved with this preservation project technically in a volunteer capacity. I can build a cutting edge resilient distributed storage system for cheaper than anything currently on the market.
Please reach out or pass along my email.
Alex
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Linas, Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic. I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction. -- Alex Gorbachev On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The intent is the US administration’s assault against science, Linas doesn’t *want* to do it, he wants to preserve for the hope of a better future.
On Apr 8, 2025, at 9:28 AM, Alex Gorbachev <ag@iss-integration.com> wrote:
Hi Linas,
Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic.
I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction.
-- Alex Gorbachev
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I was trying to analyze the original request, which seems to be something of the following set: - The goal is to archive a large amount of (presumably public) data on a community run globally sharded or distributed storage. - Can Ceph be used for this? Seems no, at least not in a sense of running lots of OSDs at different locations by different people loosely coupled into one global public data repo. Perhaps, there are some other ideas from people who have done this kind of thing. - Are there restrictions on obtaining the data? If it's public and accessible now, it should be able to be copied. If not, what are the restrictions on obtaining and copying the data? - Organization: how will the storage and maintenance of data be organized (and funded)? A foundation, a SETI-at-home like network, a blockchain (to preserve data veracity)? - Legal support? -- Alex Gorbachev On Tue, Apr 8, 2025 at 9:41 AM Anthony D'Atri <aad@dreamsnake.net> wrote:
The intent is the US administration’s assault against science, Linas doesn’t *want* to do it, he wants to preserve for the hope of a better future.
On Apr 8, 2025, at 9:28 AM, Alex Gorbachev <ag@iss-integration.com> wrote:
Hi Linas,
Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic.
I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction.
-- Alex Gorbachev
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I don't think Linus is only concerned with public data, no. The United States Government has had in places for many years effective means of preserving their data. Some of those systems may be old and creaky, granted, and not always the most efficient, but they suffice. The problem is that the current administration and their unelected henchmen are running rampant over them. Firing people with critical knowledge, and ordering the destruction of carefully amassed data. So what we're looking at is more akin to wikileaks than to simple data archival. That is, for the good of the nation - and the world - that data should be preserved, not purged. And yes, that might include obtaining information by shady means because the administration would rather see some data completely expunged in the same way that the Taliban destroyed the Buddha statues. A Ceph-like system would definitely be a good model, but for the sake of reliability and accessibility, we'd want something that could endure nodes popping in and dropping out, and to lessen the chances of a full-out invasion, that data would best be anonymously replicated over many nodes. Ceph loves very large datasets, so it should be sufficient were we to keep all our eggs in one basket. but again, that data has enemies, so extra measures are needful that Ceph wasn't designed to handle. I think such a project is doable, but it should borrow from many other community service architectures as well. It's not something that is currently available off-the-shelf, but neither were a lot of the technologies we now depend on every day. On 4/8/25 13:26, Alex Gorbachev wrote:
I was trying to analyze the original request, which seems to be something of the following set:
- The goal is to archive a large amount of (presumably public) data on a community run globally sharded or distributed storage.
- Can Ceph be used for this? Seems no, at least not in a sense of running lots of OSDs at different locations by different people loosely coupled into one global public data repo. Perhaps, there are some other ideas from people who have done this kind of thing.
- Are there restrictions on obtaining the data? If it's public and accessible now, it should be able to be copied. If not, what are the restrictions on obtaining and copying the data?
- Organization: how will the storage and maintenance of data be organized (and funded)? A foundation, a SETI-at-home like network, a blockchain (to preserve data veracity)?
- Legal support?
-- Alex Gorbachev
On Tue, Apr 8, 2025 at 9:41 AM Anthony D'Atri <aad@dreamsnake.net> wrote:
The intent is the US administration’s assault against science, Linas doesn’t *want* to do it, he wants to preserve for the hope of a better future.
On Apr 8, 2025, at 9:28 AM, Alex Gorbachev <ag@iss-integration.com> wrote: Hi Linas,
Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic.
I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction.
-- Alex Gorbachev
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Alex, "Cost concerns" is the fig leaf that is being used in many cases, but often a closer look indicates political motivations. The current US administration is actively engaged in the destruction of anything that would conflict with their view of the world. That includes health practices - especially regarding vaccination, climate data, the role of women and non-white people in history, and whatever else offends their fragile minds. For example, here in the Free State of Florida, the governor has been promoting the idea that slavery was not a bad thing because it gave forcibly-imported black people "useful job skills", textbooks must now refer to the "Gulf of America", fluoridation of water is a Bad Thing, and much more. Famous non-white people are being scrubbed from military websites and even national park webpages - and non-white, non-male people being fired from top-level government/military positions. Some are even joking that Harriet Tubman be re-classified as a "human trafficer" (this is in reference to the Underground Railroad). Which is why I don't think we should stop at just saving NIH data. Virtually all government-controlled data is at risk. And by the way, DOGE has just bragged that they saved the US government a whole million dollars by getting rid of records on magnetic tape (where they put the data afterwards wasn't said). So forget magtape archives inside the government itself. The science-fiction novel "A Canticle for Liebowitz" by Walter M. Miller outlines a post-nuclear future where the survivors rebel against knowledge, proudly bragging of being "simpletons" and burning books (and I think also educated people). Their USA counterpart is MAGA, who inherited a long history of "I don't need no librul education, I got's comun since!". Or, as Isaac Asimov put it, the idea that "my ignorance is just as good as your knowledge". This is not a concept unique to the USA, but the monkeys are firmly in charge of the zoo at this point so protecting everything we can is really important. Tim On 4/8/25 09:28, Alex Gorbachev wrote:
Hi Linas,
Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic.
I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction.
-- Alex Gorbachev
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Alex, The data purge is political. Data includes research on gun violence, sexually transmitted disease, you name it. The scientists keeping the data were keeping it for conventional science reasons. Labs are typically funded, and storage costs are paid for by "principal investigators": the big name getting the research grants, determining the research directions, hiring the junior scientists to do the work (and paying rent for the lab, buying the equipment). Since a large number of principal investigators were fired, and their labs have been dissolved, equipment sold off, the data that those labs kept is next. Some of this data is shared internationally. I've been talking to an Australian genomics researcher, they're working with the Australian govt to find funding to purchase the servers and storage needed to obtain and preserve copies of datasets. Those conversations reveal to me that it's utter chaos and confusion; no one knows anything, no one knows how to copy datasets, how much time they have to do this. A month ago, everything was fine: they used some random gnu-R or SciPy widgets that query some ncbi.nih.gov database. Now it's "where is this database and how would I copy it?" types of questions/conversations. Chaos. These are biologists, not programmers, not system admins. No visibility into the network architecture. No idea ***at all***. Like all things chaotic, it's difficult to discern what the actual problem is, what the danger is, what the timescale is, who's doing what. As to the scale: something like 20K or 50K workers at NIH were fired. I cannot keep track of the headlines. Headlines include reports that lab animals, lab mice were just abandoned and have died of thirst. Was this one lab? A dozen labs? Is this some fake internet news report? I can't tell. But it's a thermometer for gauging the rapidity of the shutdown: not even orderly enough to deal with cleaning out the labs before turning off the lights. As to tape backup: I am clearly more online than you are, because just a few days ago, DOGE announced that they "saved" almost $1M (That's M as in million -- six zeros) of "unnecessary and wasteful expenditures" by destroying a tape library with 75K tapes in it. These are the types of headlines circulating around. Politically, this is the destruction of health and biology research in America. Will the Europeans pick up the slack? Who knows. The short-term tactical issue is about the preservation of datasets. The long-term strategic issue is designing storage solutions that are robust against political attack. The short-term situation is total chaos, and I understand nothing at all about the status. The long-term strategic issue is one I've been keeping an eye on for decades, and is what led me to Ceph. -- Linas On Tue, Apr 8, 2025 at 8:28 AM Alex Gorbachev <ag@iss-integration.com> wrote:
Hi Linas,
Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic.
I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction.
-- Alex Gorbachev
On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us.
If you have a line on the data, I have connections that can store it or I can consult pro bono on building a system to store it. However wan-ceph is not the answer here. On Sun, Apr 6, 2025, 11:08 PM Linas Vepstas <linasvepstas@gmail.com> wrote:
OK what you will read below might sound insane but I am obliged to ask.
There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere.
I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know.
While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list.
OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment.
Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope?
Am I misunderstanding something about IPFS that merits taking a second look at it?
Is there any other way of getting scalable reliable "decentralized" internet-wide storage?
I mean, yes, of course, the conventional answer is that it could be copied to AWS or some national lab or two somewhere in the EU or Aus or UK or where-ever, That's the "obvious" answer. I'm looking for a non-obvious answer, an IPFS-like thing, but one that actually works. Could it work?
-- Linas
-- Patrick: Are they laughing at us? Sponge Bob: No, Patrick, they are laughing next to us. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (9)
-
Alex Buie
-
Alex Gorbachev
-
Anthony D'Atri
-
Anthony D'Atri
-
Laimis Juzeliūnas
-
Linas Vepstas
-
Robert Sander
-
Tim Holloway
-
Šarūnas Burdulis