Hi Dominik, I assume that you are talking about a cephfs problem. To identify the root cause you have to debug the log file of the mds servers. Joachim joachim.kraftmayer@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Am Sa., 14. Sept. 2024 um 14:24 Uhr schrieb dominik.baack < dominik.baack@cs.tu-dortmund.de>:
Hi, we started upgrading our Ceph Cluster consisting of 7 Nodes from quincy to reef two days ago. This included the upgrade of the underlying OS and several other small changes. After hitting the osd_remove_queue bug, we could recover mostly, but are still in a non-healthy state because of changes on our network. (See attached image) But overall we can mount the filesystem on all nodes, read and write.
The Problem now is that at least one file created by slurmctld exist which seems to be somehow compromised during this. It cannot be read or removed on any of the storage or working nodes. All operations (cat, rm, less, ...) are stuck until they are forcefully terminated. I checked all nodes manually and could not identify any process with an open handle to the file.
If possible we would like to unblock the file, but removing would also be a possiblity.
Cheers Dominik _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io