It has come to my attention there are outstanding questions about what should be working, what’s not, timelines, etc.  I’ll do my best to address the questions and send continued and more frequent updates.

To Recap

The “Sepia” lab was shut down on December 15 and most of the hardware was relocated to an IBM datacenter in Poughkeepsie, NY.  It arrived on December 19 and is in the process of being re-racked.  Power is in place under the floors already.  Rack elevations are figured out.  The lab admins are just in the process of re-racking and re-cabling which is going to take some time.  The folio are re-racked and cabled.  robsoni, ivan, braggi, and adami are in progress.  smithi and vossi are next.  Based on my experience with IBM getting the Octo lab’s Red Hat switches online in Tucson, I estimate another month.

Moved: adami, braggi, confusa, folio, gibba, ivan, mako, officinalis, reesi, robsoni, smithi, sulcata, vossi, ssdstore, gitbuilder, hv, cnv

NOT MOVED: incerta, irvingi, mira, omani, refarch, senta

We aren’t sure yet.  We have observed if a VPN client rsyncs data over the tunnel, all VPN clients (including the rsync..er) start to experience >=50% packet loss.  We need debugging time to narrow down the bottleneck and engage IBM Networking and/or the ISP and/or IBM Security and/or Palo Alto Networks.  In the meantime, Adam on my team is setting up a proof of concept for https://goteleport.com/.   What I can say is it’s very unlikely we will reuse the exact same OpenVPN configuration we used for the RDU lab.

My PR Checks / Jenkins builds are taking forever

Yes, we went from ~30 bare metal x86 builders to 8.  Initially, in December when the RDU lab first shut down, it appeared those 8 were enough to keep up with the workload but January has been much busier than anticipated.  The Ceph Windows PR check failures added to the load because they kept failing and created a long backlog.  They also take the most amount of time to run by far.  Roughly 2 hours per PR, whereas actual package and container builds are under an hour.  “make check” is under an hour.  Adam K is also spinning up some additional baremetal builders today to help with the load.

Can I push to ceph-ci?

Yes, and you’ll even get a build.  Eventually.

Great, can I test it using teuthology?

Not yet.  As I hope you’ve already heard, we decided to switch to using Canonical’s Metal-As-A-Service (MaaS) for deploying OSes.  Unfortunately, we were not given enough time to deploy it, along with all the requisite services (i.e., pulpito, paddles, teuthology, LRC), and get it working with the new testnodes before the Sepia lab shut down.  So we’re figuring out the quirks now.  The main issue I have been fighting for weeks is MaaS can not handle the burstiness of teuthology job scheduling.  Essentially, one server deployment holds a lock on a postgres database, and enough database transactions pile up waiting for that lock to the point that subsequent API calls can eventually time out.  So, if we schedule 30 deployments at the same time, and another machine is already PXE booting and cloud-init is asking MaaS for its kickstart/preseed, it occasionally times out, the deployment fails, and the teuthology job dies.

I am deciding we have spent enough time spinning wheels and am in the process of deploying FOG as an interim measure to get us unblocked.  Either additional teuthology development will be required to throttle deployments (which I don’t think it a reasonable ask), or we will need assistance from Canonical with no guaranteed timeline for resolution.

Are teuthology job logs available to be analyzed?

Logs for jobs ran prior to December 15 are still on the old LRC which needs to be re-racked, re-cabled, and switches reconfigured still.  Logs for new teuthology runs will be on the new LRC.  If you want the new LRC mounted to your dev box, let me know.  It requires installing ceph-common and I don’t want to interfere with existing packaging/repos/etc. without your prior approval.

I need a Dev Box

“sockeni” are the new dev boxes.  Look at https://wiki.sepia.ceph.com/doku.php?id=hardware:sockeni#users.  Based on which team members you most closely align with, let me know, and I’ll add your user and SSH key.  Sharing is caring for now.  We’ll look at sharing, spreading things out, etc. once the Red Hat gear is brought back online.

Additional Perspective

There are still over 200 servers in Tucson that are unusable due to a combination of PTO, switch port misconfigs, last minute subnet/architecture changes, MaaS growing pains, repeated power outages, etc. so my team is still spread really thin.

Let me know if you have any questions.