Apparently despite using the —preserve_setting flag with Supermicro’s update utility, the boot order got reset on all of the nodes so all of the jobs that just got picked up will die.  Dan is working on fixing the boot order again.  I stopped the dispatcher until this is done.

From: David Galloway <David.Galloway@ibm.com>
Date: Monday, February 9, 2026 at 8:26 PM
To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io>
Subject: Re: Queue is Paused

I’ve restarted the dispatcher on soko04.  All of the trial testnodes have the latest BIOS and I verified all of the up and unlocked testnodes reimaged fine with FOG.

Scheduled jobs will continue to be processed now.  The hope is this will resolve all of the deployment failures e.g., https://tracker.ceph.com/issues/74774 https://tracker.ceph.com/issues/74717.  We are unable to manually reproduce those conditions and we have checked a testnode when it gets left in that condition and we’re unable to glean any useful information so we are just having to hope that a BIOS update will take care of it.

From: David Galloway <David.Galloway@ibm.com>
Date: Monday, February 9, 2026 at 12:51 PM
To: dev <dev@ceph.io>, David Galloway via Sepia <sepia@ceph.io>
Subject: Queue is Paused

I’ve paused the teuthology queue so we can update the firmware on the new trial nodes in an attempt to resolve some of the Dead/Fail jobs.

-- 

David Galloway

Ceph Engineering Labs – Infrastructure Architect

david.galloway@ibm.com

IBM