PREX Troubleshooting

last updated: Apr 25, 2010

What to do if...

Temperatures from Collimator: no readout

Need to reboot the IOC that controls the temperature readings.

Need to reboot
iocha12 (129.57.188.11) - Beamline slow controls

To verify it's booting, you can (optionally) login to the serial port:
serial port on "telnet 129.57.188.87 2007" or "telnet hatsv12 2007"

Can do the "soft" reboot (backplane reset) :
Press the Green button 8 in Reset Box #2 in middle room of Counting Room
located located in bottom of rack CH01B04

Hold the green button till the count of five, it is a long cable to go to
the IOC, lots of capacitance.

See also
IOC locations

Target Computer Frozen

If you are not sure, turn off beam, call the target expert. The present target control computer (the monitor is labelled "POLTARAC"; the CPU sits underneath the monitor behind the door at your feet; it is labelled "POL-TAR"). If the target control windows freeze, and does not respond to keyboard/mouse input, then < you can open the door at your feet and power off/on the computer. The computer will restart, and will not ask you for a username or password. You should open a terminal window and you should see that you are in the "poltar" account. Then do the following:
cd Screens
./tgtgui
and the target control GUIs should start.

BPM12 is saturating

N.B Instructions modified 8 Sep 2009 to account for 18 bit ADC.

During data taking keep an eye on the BPM12 stripcharts in Panguin. All channels should be roughly 85K. If they go to around 230K they are saturating. This is at present regarded as highly unlikely if not impossible, however data with BPM12 saturating is unacceptable; beam might as well be off. In fact, when BPM12 is saturating we log this as "Beam not acceptable". (Exception: BPM saturation in the first minute or so after a beam trip -- but not beyond that -- is harmless.)

If BPM12 is saturating IMMEDIATELY do the following:

  1. Call MCC and request they do "gain search" on FFB.
  2. Also ask MCC to check that FFB is running in "manual gain" mode, rather than the FFB "auto gain" mode on the bpms. (The control for this is found under the debug tab, on the FFB control screen, under the title "BPM gain control").
  3. If that doesn't clear the problem, ask them to:
    1. Turn FFB off.
    2. put the crate in SEE normal ops and wait for BPM12 to return to normal gain on our Panguin monitor.
    3. Turn FFB back on, in manual gain mode.

If none of this works call the RC, and MCC should call their software on call people.

Log the problem in HALOG. When filling out the beam accounting, report "Beam Not Acceptable" for the duration of the time BPM12 was saturating.

Empty Histograms?!?!


WHY? SEVERITY HOW TO CHECK? WHAT TO DO?
No Q2 plots, or anything else that needs tracks. Okay These plots are only filled for special runs VDCs are off at high current. Do NOT turn on.
Beam Current is
    less than 95 A
Okay Check "Hall A Current Monitor"
Wait/Ask for higher beam current.
Beam Is Tripping Frequently
(More than 1 trip every
     5 minutes)
Not So Good Check "Hall A Current Monitor"
Wait for stable current, or
    ask MCC for better stability.
Beam Is Noisy Not Good Difficult to diagnose
Ask MCC to check "Laser Phase Noise"
    or "Injector Transmission"
Ask MCC to fix it.
    (Source On-Call or Optics On-Call)
PREX Detectors
     are saturated
VERY BAD Check "Detector Stripcharts"
    in Online Monitor
Vertical scale should show signal less than 50k
Adjust detector HV to bring signal below 45k
Page an expert for help
BPMs are saturated VERY BAD Check "Target BPMs" and "ARC BPMs"
    in Online Monitor
Horizontal scale for each BPM
     should be the same
Events will be cut if the BPM is saturated.
Inform MCC that BPM gains are too high.
    iocse9 for BPM12
    iocse10 for BPM 4A and 4B
Page an expert for help


  • VDC HV cannot turn on.:
    
    Solution: go to http://hareboot22.  User/password are on the 
    paper near the whiteboard.
    
    Reboot iocha22 and wait 5 minutes.  Might have to restart
    the Tools GUI (usually not).
    
    
    
  • You lose network connectivity:
  • For all of the above network problems, do the following to power cycle the respective network switch:

  • You lose a magnet controls IOC for an HRS:

  • [The readbacks for either or both of the HRS magnets go white on the "Hall A Tools". Q1 will probably still read, it is on another IOC]

    For the Left HRS:

    For the Right HRS:

  • HV EPICS for Beamline or L-HRS are white
    Note, the R-HRS EPICS HV is semi-permanently white. Don't worry about it. See the opmanual for instructions on adjusting HV.
    If Beamline and/or L-HRS EPICS screen for HV shows white fields, you can reboot iocha11 and iocha22 via http://hareboot22. The user/password are on the "passwords" paper near the white board.

  • The Cavity Monitors stop working or saturate:

  • [There is beam, but bcmcav2 and/or bcmcav3 are close to zero ADC channels or negative. Or saturating the ADCs.]

    Reset the Cavity Crate:

    Setting Gains for both cavities. Judge from raw ADC if the gains are ok.

    See the screen shots from Fall 2009
    
    
    
    
    

  • Prex CODA config is hung up, ROC31 not reporting status

  • NOTE: Only CAPS Y will work, y wont work!!