Don't laugh. I am experimenting with Ceph in an enthusiast,
Everyone has a smile on their face when working with ceph! ;)
Seriously, I think that, with just a little bit of polishing and automation, Ceph could be deployed in the small-office/home-office setting. Don't laugh. This could happen.
I am also looking at this now. Some offices have 30-70 drives. Then it starts to make sense I guess. Currently exploring individual performance.
wtf? I cd to /dest/.snap and indeed it is a directory. And it's empty. The source .snap is a file, 736 bytes in size. Directory timestamp and source-file timestamp are identical. I attempt a hand repair. I cd to the parent dir, do rmdir .snap and rmdir says "no such directory". wtf? So then `ls -la` and indeed, there is no such directory. But I can still cd into it! Huh. I cannot rmdir it, I cannot cp or mv it, because "it doesn't exist". If I cd into it, I cannot create any files in it, I get "permission denied", even as root. So it's a directory, it won't show in the listing of the parent dir. I can cd to it, but I cannot put anything in it.
Did you read the manuals about cephfs snapshots? I skipped a bit through the rest of your message. You should know this a quite capable team here at ceph and it is very unlikely you have anything to touch them ;) Get to know ceph. You think companies using 3000 disks would use ceph having the issues you mention?