The first thing we noticed was that the right Q3 momentum readback didn't change when the P0 setpoint was set to 442 MeV/c. We later notice that the Q3 Hall probe readback had been dead since Monday afternoon.
Paged John LeRose, then Javier without luck. Got a hold of Jack and he recommended power cycling the Lakeshore Gaussmeter through hareboot27. No effect. Tried rebooting the RHRS magnet control IOC (iocha16) next. That did not solve the Q3 readback problem either. At this point the ARM was here for an escorted Controlled Access so I went down and reseated the serial connector on the back of the Lakeshore unit. That fixed the Hall probe readback.
Shortly after the IOC reboot we noticed that the Q1 LHe "L(%)" readback on the Hall A Tools GUI was now reading "inf". The Helium levels GUI reported a good level and was updating so we decided to must monitor the level using that GUI and deal with the Tools display tomorrow.
We came back up to the Counting House to see that the right dipole was ramping down. Jack pulled up some stripcharts and it seems the dipole actually started ramping down BEFORE we reset the right IOC and we just didn't notice it at the time. The dipole power supply gui showed a Quench fault (no screen snap, sorry). After it finished ramping down we reset the supply through the GUI (seemed to work) and tried to ramp to 30A as a test. It didn't respond immediately but after a minute did start to ramp. Unfortunately it settled into a nonsensical state: see 210743. It stopped ramping by itself before reaching 30A and started to (very) slowly ramp down. Jack went down to the Hall to power cycle the Allan Bradley, Octogon, and Dynapower supplies. There was were two faults on the supply that would not clear: one labeled "FC Fault", and a second associated with a thermally controlled resistor (can't remember the 4 letter label...) No amount of swearing or cycling would clear those faults so we threw in the towel until morning.
It occurs to me that I also powercycled the "GPIB devices (magnet current
readback DVMs)" outlet through hareboot25 when discussing the Q3 hall
probe problem with Jack. Could that have triggered any of the later
problems with the dipole?