The problem turns out to be burning the candle at both ends. I have been checking network communication for the past few hours and haven't realized I was using my 1Gb IPs, not the 100Gb IPs. The 100Gb got connected to the wrong ports on the cable move. Thanks for the attempted assists. Focusing on the mons at least eventually lead to finding the error. -Chip On Thu, Feb 22, 2024 at 7:26 AM Schweiss, Chip <chip@innovates.com> wrote:
I had to temporarily disconnect the network on my entire Ceph cluster, so I prepared the cluster by following what appears to be some incomplete advice.
I did the following before disconnecting the network: #ceph osd set noout #ceph osd set norecover #ceph osd set norebalance #ceph osd set nobackfill #ceph osd set nodown #ceph osd set pause
Now, all the ceph services are still running, but I cannot undo any flags:
root@proxmox01:~# ceph osd unset pause 2024-02-22T13:16:02.220+0000 7f0aab5a26c0 0 monclient(hunting): authenticate timed out after 300
[errno 110] RADOS timed out (error connecting to the cluster)
Any advice on how to recover would be greatly appreciated.
Thank you, -Chip