Hi ! Have you mounted cephfs with the 'root_squash' option set? If so, remove that option. I may be wrong here, but as far as I know, this is still considered unsafe and can lead to data corruption since the necessary code changes have not yet made it into the mainstream linux kernel. Carsten ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze@cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze
Am 12.02.2026 um 07:15 schrieb Dominik Baack via ceph-users <ceph-users@ceph.io>:
Hi,
I noticed now that files are still actively corrupted / replaced by empty files when open saved.
Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. Ceph servers are still running 20.2 Tentacle (deployed via cephadm) with an Ubuntu 24.04 host.
Options currently set are to noout, norebalance, noscrub and nodeep-scrub
I am currently setting up a read only mount to copy all existing data for a backup.
I have currently no clue whats going on, I currently was able to observe this behavior only on the nodes. As those were upgraded as well (MLNX driver, nvidia-fs,...) can it be network?
Any Idea how to recover from this?
Cheers Dominik
Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users:
Hi,
after an controlled shutdown of the whole cluster do to external circumstances we decided to update from 19.2 to 20.2 after the restart. The system was health before and after the update. The nodes mounting the filesystem were not equally lucky and were partially shutdown hard. Storage was kept running additional ~30min after node shutdown, all inflight operations should have finished.
Now we discover the some of the user files seem to be replaced with zeros. For example:
stat .gitignore File: .gitignore Size: 4429 Blocks: 9 IO Block: 4194304 regular file Device: 0,48 Inode: 1100241384598 Links: 1
hexdump -C .gitignore 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| 0000114d
Scanning for files containing only zeros show several issues of files that were likely accessed before or during the shutdown of the nodes.
How should I progress from here?
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io