HAPPEX/PVDIS Troubleshooting

What to do if...

• BPM12 is saturating
• Empty Histograms
• You lose network connectivity
• You lose a magnet controls IOC for an HRS
• Cavity Monitors stop working
• Happex CODA config is hung up, ROC31 not reporting status

N.B Instructions modified 8 Sep 2009 to account for 18 bit ADC.

During data taking keep an eye on the BPM12 stripcharts in Panguin. All channels should be roughly 85K. If they go to around 230K they are saturating. This is at present regarded as highly unlikely if not impossible, however data with BPM12 saturating is unacceptable; beam might as well be off. In fact, when BPM12 is saturating we log this as "Beam not acceptable". (Exception: BPM saturation in the first minute or so after a beam trip -- but not beyond that -- is harmless.)

If BPM12 is saturating IMMEDIATELY do the following:

1. Call MCC and request they do "gain search" on FFB.
2. Also ask MCC to check that FFB is running in "manual gain" mode, rather than the FFB "auto gain" mode on the bpms. (The control for this is found under the debug tab, on the FFB control screen, under the title "BPM gain control").
3. If that doesn't clear the problem, ask them to:
   1. Turn FFB off.
   2. put the crate in SEE normal ops and wait for BPM12 to return to normal gain on our Panguin monitor.
   3. Turn FFB back on, in manual gain mode.

If none of this works call the RC, and MCC should call their software on call people.

Log the problem in HALOG. When filling out the beam accounting, report "Beam Not Acceptable" for the duration of the time BPM12 was saturating.

• You lose network connectivity:
  
• 192 Subnet: halladaq-cat [You cannot ping the crates in the hall, e.g. for ROC25(L-arm) or ROC26(R-arm) you cannot ping the respective IP (happex8, happex7). Also cannot ping spectrometer DAQ crates like hallasfi4]
• 193 Subnet: compton [You cannot ping the any Compton crate, cannot reboot them.]
• Control crates [Loss of network connectivity to controls IOCs]
For all of the above network problems, do the following to power cycle the respective network switch:
• Goto https://halla-pwr/
• Accept the certificate if asked.
• Username: halla-pwr
• Password: see the sheet near the white board in the counting room.
• Click "Power Management" on the left.
• You will see 9 outlets. Three are currently used:
  • halladaq (192 DAQ subnet) -- full name is halladaq-cat
  • hac-cat -- full name is hac-cat3550
  • compton -- full name is compton-cat2950
• There is a bulb icon in column "outlet status". If the bulb is yellow the outlet is on. To power cycle the outlet, first turn it off by clicking the bulb (it will change to grey), wait a few seconds, then turn it back on again. This on/off will power cycle the network switch.
• You lose a magnet controls IOC for an HRS:

[The readbacks for either or both of the HRS magnets go white on the "Hall A Tools". Q1 will probably still read, it is on another IOC]

For the Left HRS:

• Click the IOC's GUI (from the "Hall A Main Menu")
• You will see that the iocha14 is whited out (don't worry about the "Rich detector" and "Waterfall Target" iocs being also whited out, we don't use them for HAPPEX). If it is not whited out, you have some other problem.
• Under "Reboot" you will see the URL for the reboot for that IOC, in this case http://hareboot10.
• Go to a WWW browser, type in http://hareboot10 as the address, login: the username and password are both written on the piece of paper on the wall near the whiteboard.
• Select "Immediate Reboot", under "Control Action", for the outlet marked IOCHA14.
• Click "Apply"
• The green "ON" under State should changed to OFF.
• Wait a few minutes, then the readout for the magnets should reappear in Hall A Tools.
• Check the state of the magnets; if they are not ON and at the correct currents, then do the following:
  
  From the HallA General Tools window, there is a P0 (central momentum) SET field for each spectrometer in which you can eneter the desired momentum setting. To change to a lower momentum from a higher previous setting, you just need to enter the momentum value, no worry about cycling the quads. To bring momentum setting up, you need to cycle the quads Q2 and Q3 of either arm. That is increasing the current of Left Q2/Q3 to 1600 A or 1400 for Right arm, wait for 2 minutes after they reach max, then tuning down the value to the desired setting. More detailed instructions are on the white board of the counting house. There's no need to cycle Q1, Dipole.

For the Right HRS:

• Go to the middle room, and look for the green button labelled "Right Spectrometer Controls VME"; it is at the lower right of the lower box of green reset buttons in one of the middle racks.
• Push the button, holding it for 3-4 seconds.
• Wait a few minutes, then the readout for the magnets should reappear in Hall A Tools.
• Check the state of the magnets; if they are not ON and at the correct currents, then do the same process as explained above for the left HRS.
• The Cavity Monitors stop working or saturate:

[There is beam, but bcmcav2 and/or bcmcav3 are close to zero ADC channels or negative. Or saturating the ADCs.]

Reset the Cavity Crate:

• Page Zafar Ahmed, Xiaoyan Deng, Bob Michaels, or John Musson in that order.
• At the moment the remote power-cycle does not work. May need to fix this. If not, access is required.
• Using your ops account, ssh to opsll1
• Type "menubar"
• A small menubar box appears. Click on this. Click on "EDM" (2nd choice at the moment)
• You will see 2 windows. You care about the edm 1-11-11zm (small window)
• Click "File" -> "Open" -> go to directory Happex_Bpm
• Double click on the directory
• Click on HxBpm2a.edl. And click OK.
• 3H02 is the upstream monitor. What we call cavity2
• Click on HxBpm2b.edl. And click OK.
• 3H03 is downstream monitor. What we call cavity3
• After changing values, click "Save & Restore". A small dialog pops up. Select "Save"
• A window opens. Enter (carriage return), wait, then enter again.
• When changing any pre-gain make sure the Intensity of Positin are below 32,000.

Setting Gains for both cavities. Judge from raw ADC if the gains are ok.

• Set BCM Pre Gain to 4.
• Set BCM Post Gain to 0
• Set X Pre Gain to 2 for 3H02 and 1 for 3H03
• Set X Post Gain to 0
• Set Y Pre Gain to 2 for 3H02 and 1 for 3H03.
• Set Y Post Gain to 0.
See the screen shots from Fall 2009


• Pvdis CODA config is hung up, ROC31 not reporting status




• Find the apar@adaql1 terminal, or open one, and kill the runControl by typing

  > kcoda

• If you get a warning that someone else might be using coda, and whether or not you really want to kill coda, type

  > Y

NOTE: Only CAPS Y will work, y wont work!!


• Reboot ROC31s   Easiest way is to power cycle at g0injereboot   (chan 1,2,3 are resp. g0inj, happexinj, and hatsv18. See passwd on paper near whiteboard.)
• After the ROC has rebooted successfully, bring up the runControl by typing,

  > startcoda

• Again, if you are warned of someone else using coda, and whether or not you want to startcoda, type

  > Y

• The runControl gui should appear. On the runControl gui, click

  > Connect

• Click on Configure, and select Pvdis
• After the configuration has been set properly, which can be checked by observing the bottom window in the runControl, click

  > Download

• Wait utill you see Download Succeded message at the bottom window of the runControl gui. If instead, you see ROC31 not reporting status for # sec, then you need to kill the runControl, and reboot ROC 31 again.

  Rebooting ROC31

  • Check the terminals to see if we are logged on to ROC31 already, which we should be.
  • If we are not logged on to ROC31 already, login from any terminal as

    > rlogin g0inj

  • Reboot the ROC31 crate by typing

    > reboot

  • Once the reboot starts, you will be logged out automatically, which is normal. It takes about 30 sec or so, no longer than 1 minute for the crate to reboot.

  PVDIS How To Page

  ---

  If you still can't find a resolution contact:

  whoever is maintaining this page
  Last modified: TBP