PREX Troubleshooting

What to do if...

• Target Computer Frozen
• Temperatures from Collimator not reading (EPICS)
• BPM12 is saturating
• Empty Histograms
• Rebooting VDC (EPICS stuck)
• You lose network connectivity
• You lose a magnet controls IOC for an HRS
• HV screens white (beamline or L-HRS EPICS)
• Cavity Monitors stop working
• Prex CODA config is hung up, ROC31 not reporting status

Need to reboot the IOC that controls the temperature readings.

Need to reboot
iocha12 (129.57.188.11) - Beamline slow controls

To verify it's booting, you can (optionally) login to the serial port:
serial port on "telnet 129.57.188.87 2007" or "telnet hatsv12 2007"

Can do the "soft" reboot (backplane reset) :
Press the Green button 8 in Reset Box #2 in middle room of Counting Room
located located in bottom of rack CH01B04

Hold the green button till the count of five, it is a long cable to go to
the IOC, lots of capacitance.

See also
IOC locations

cd Screens
./tgtgui

N.B Instructions modified 8 Sep 2009 to account for 18 bit ADC.

During data taking keep an eye on the BPM12 stripcharts in Panguin. All channels should be roughly 85K. If they go to around 230K they are saturating. This is at present regarded as highly unlikely if not impossible, however data with BPM12 saturating is unacceptable; beam might as well be off. In fact, when BPM12 is saturating we log this as "Beam not acceptable". (Exception: BPM saturation in the first minute or so after a beam trip -- but not beyond that -- is harmless.)

If BPM12 is saturating IMMEDIATELY do the following:

1. Call MCC and request they do "gain search" on FFB.
2. Also ask MCC to check that FFB is running in "manual gain" mode, rather than the FFB "auto gain" mode on the bpms. (The control for this is found under the debug tab, on the FFB control screen, under the title "BPM gain control").
3. If that doesn't clear the problem, ask them to:
   1. Turn FFB off.
   2. put the crate in SEE normal ops and wait for BPM12 to return to normal gain on our Panguin monitor.
   3. Turn FFB back on, in manual gain mode.

If none of this works call the RC, and MCC should call their software on call people.

Log the problem in HALOG. When filling out the beam accounting, report "Beam Not Acceptable" for the duration of the time BPM12 was saturating.

Solution: go to http://hareboot22.  User/password are on the 
paper near the whiteboard.

Reboot iocha22 and wait 5 minutes.  Might have to restart
the Tools GUI (usually not).

• 192 Subnet: halladaq-cat [You cannot ping the crates in the hall, e.g. for ROC25(L-arm) or ROC26(R-arm) you cannot ping the respective IP (happex8, happex7). Also cannot ping spectrometer DAQ crates like hallasfi4]
• 193 Subnet: compton [You cannot ping the any Compton crate, cannot reboot them.]
• Control crates [Loss of network connectivity to controls IOCs]

• Goto https://halla-pwr/
• Accept the certificate if asked.
• Username: halla-pwr
• Password: see the sheet near the white board in the counting room.
• Click "Power Management" on the left.
• You will see 9 outlets. Three are currently used:
  • halladaq (192 DAQ subnet) -- full name is halladaq-cat
  • hac-cat -- full name is hac-cat3550
  • compton -- full name is compton-cat2950
• There is a bulb icon in column "outlet status". If the bulb is yellow the outlet is on. To power cycle the outlet, first turn it off by clicking the bulb (it will change to grey), wait a few seconds, then turn it back on again. This on/off will power cycle the network switch.

For the Left HRS:

• Click the IOC's GUI (from the "Hall A Main Menu")
• You will see that the iocha14 is whited out (don't worry about the "Rich detector" and "Waterfall Target" iocs being also whited out, we don't use them for PREX). If it is not whited out, you have some other problem.
• Under "Reboot" you will see the URL for the reboot for that IOC, in this case http://hareboot10.
• Go to a WWW browser, type in http://hareboot10 as the address, login: the username and password are both written on the piece of paper on the wall near the whiteboard.
• Select "Immediate Reboot", under "Control Action", for the outlet marked IOCHA14.
• Click "Apply"
• The green "ON" under State should changed to OFF.
• Wait a few minutes, then the readout for the magnets should reappear in Hall A Tools.
• Check the state of the magnets; if they are not ON and at the correct currents, then do the following:
  
  From the HallA General Tools window, there is a P0 (central momentum) SET field for each spectrometer in which you can eneter the desired momentum setting. To change to a lower momentum from a higher previous setting, you just need to enter the momentum value, no worry about cycling the quads. To bring momentum setting up, you need to cycle the quads Q2 and Q3 of either arm. That is increasing the current of Left Q2/Q3 to 1600 A or 1400 for Right arm, wait for 2 minutes after they reach max, then tuning down the value to the desired setting. More detailed instructions are on the white board of the counting house. There's no need to cycle Q1, Dipole.

For the Right HRS:

• Go to the middle room, and look for the green button labelled "Right Spectrometer Controls VME"; it is at the lower right of the lower box of green reset buttons in one of the middle racks.
• Push the button, holding it for 3-4 seconds.
• Wait a few minutes, then the readout for the magnets should reappear in Hall A Tools.
• Check the state of the magnets; if they are not ON and at the correct currents, then do the same process as explained above for the left HRS.

Reset the Cavity Crate:

• Page Zafar Ahmed, Bob Michaels, or John Musson in that order.
• At the moment the remote power-cycle does not work. May need to fix this. If not, access is required.
• Using your ops account, ssh to opsll1
• Type "menubar"
• A small menubar box appears. Click on this. Click on "EDM" (2nd choice at the moment)
• You will see 2 windows. You care about the edm 1-11-11zm (small window)
• Click "File" -> "Open" -> go to directory Happex_Bpm
• Double click on the directory
• Click on HxBpm2a.edl. And click OK.
• 3H02 is the upstream monitor. What we call cavity2
• Click on HxBpm2b.edl. And click OK.
• 3H03 is downstream monitor. What we call cavity3
• After changing values, click "Save & Restore". A small dialog pops up. Select "Save"
• A window opens. Enter (carriage return), wait, then enter again.
• When changing any pre-gain make sure the Intensity of Positin are below 32,000.

Setting Gains for both cavities. Judge from raw ADC if the gains are ok.

• Set BCM Pre Gain to 4.
• Set BCM Post Gain to 0
• Set X Pre Gain to 2 for 3H02 and 1 for 3H03
• Set X Post Gain to 0
• Set Y Pre Gain to 2 for 3H02 and 1 for 3H03.
• Set Y Post Gain to 0.

• Find the apar@adaql1 terminal, or open one, and kill the runControl by typing

  > kcoda

• If you get a warning that someone else might be using coda, and whether or not you really want to kill coda, type

  > Y

• Reboot ROC31s   Easiest way is to power cycle at g0injereboot   (chan 1,2,3 are resp. g0inj, happexinj, and hatsv18. See passwd on paper near whiteboard.)
• After the ROC has rebooted successfully, bring up the runControl by typing,

  > startcoda

• Again, if you are warned of someone else using coda, and whether or not you want to startcoda, type

  > Y

• The runControl gui should appear. On the runControl gui, click

  > Connect

• Click on Configure, and select Prex
• After the configuration has been set properly, which can be checked by observing the bottom window in the runControl, click

  > Download

• Wait utill you see Download Succeded message at the bottom window of the runControl gui. If instead, you see ROC31 not reporting status for # sec, then you need to kill the runControl, and reboot ROC 31 again.

  Rebooting ROC31

  • Check the terminals to see if we are logged on to ROC31 already, which we should be.
  • If we are not logged on to ROC31 already, login from any terminal as

    > rlogin g0inj

  • Reboot the ROC31 crate by typing

    > reboot

  • Once the reboot starts, you will be logged out automatically, which is normal. It takes about 30 sec or so, no longer than 1 minute for the crate to reboot.

  PREX How To Page

  ---

  If you still can't find a resolution contact:

  whoever is maintaining this page
  Last modified: TBP