HAPPEX/PVDIS Troubleshooting

What to do if...

• BPM12 is saturating
• Empty Histograms
• You lose network connectivity
• You lose a magnet controls IOC for an HRS
• Cavity Monitors stop working
• Happex CODA config is hung up, ROC31 not reporting status

N.B Instructions modified 8 Sep 2009 to account for 18 bit ADC.

During data taking keep an eye on the BPM12 stripcharts in Panguin. All channels should be roughly 85K. If they go to around 230K they are saturating. This is at present regarded as highly unlikely if not impossible, however data with BPM12 saturating is unacceptable; beam might as well be off. In fact, when BPM12 is saturating we log this as "Beam not acceptable". (Exception: BPM saturation in the first minute or so after a beam trip -- but not beyond that -- is harmless.)

If BPM12 is saturating IMMEDIATELY do the following:

1. Call MCC and request they do "gain search" on FFB.
2. Also ask MCC to check that FFB is running in "manual gain" mode, rather than the FFB "auto gain" mode on the bpms. (The control for this is found under the debug tab, on the FFB control screen, under the title "BPM gain control").
3. If that doesn't clear the problem, ask them to:
   1. Turn FFB off.
   2. put the crate in SEE normal ops and wait for BPM12 to return to normal gain on our Panguin monitor.
   3. Turn FFB back on, in manual gain mode.

If none of this works call the RC, and MCC should call their software on call people.

Log the problem in HALOG. When filling out the beam accounting, report "Beam Not Acceptable" for the duration of the time BPM12 was saturating.

• You lose network connectivity:
  
• 192 Subnet: halladaq-cat [You cannot ping the crates in the hall, e.g. for ROC25(L-arm) or ROC26(R-arm) you cannot ping the respective IP (happex8, happex7). Also cannot ping spectrometer DAQ crates like hallasfi4]
• 193 Subnet: compton [You cannot ping the any Compton crate, cannot reboot them.]
• Control crates [Loss of network connectivity to controls IOCs]
For all of the above network problems, do the following to power cycle the respective network switch:
• Goto https://halla-pwr/
• Accept the certificate if asked.
• Username: halla-pwr
• Password: see the sheet near the white board in the counting room.
• Click "Power Management" on the left.
• You will see 9 outlets. Three are currently used:
  • halladaq (192 DAQ subnet) -- full name is halladaq-cat
  • hac-cat -- full name is hac-cat3550
  • compton -- full name is compton-cat2950
• There is a bulb icon in column "outlet status". If the bulb is yellow the outlet is on. To power cycle the outlet, first turn it off by clicking the bulb (it will change to grey), wait a few seconds, then turn it back on again. This on/off will power cycle the network switch.
• You lose a magnet controls IOC for an HRS:

[The readbacks for either or both of the HRS magnets go white on the "Hall A Tools". Q1 will probably still read, it is on another IOC]

For the Left HRS:

• Click the IOC's GUI (from the "Hall A Main Menu")
• You will see that the iocha14 is whited out (don't worry about the "Rich detector" and "Waterfall Target" iocs being also whited out, we don't use them for HAPPEX). If it is not whited out, you have some other problem.
• Under "Reboot" you will see the URL for the reboot for that IOC, in this case http://hareboot10.
• Go to a WWW browser, type in http://hareboot10 as the address, login: the username and password are both written on the piece of paper on the wall near the whiteboard.
• Select "Immediate Reboot", under "Control Action", for the outlet marked IOCHA14.
• Click "Apply"
• The green "ON" under State should changed to OFF.
• Wait a few minutes, then the readout for the magnets should reappear in Hall A Tools.
• Check the state of the magnets; if they are not ON and at the correct currents, then do the following:
  
  From the HallA General Tools window, there is a P0 (central momentum) SET field for each spectrometer in which you can eneter the desired momentum setting. To change to a lower momentum from a higher previous setting, you just need to enter the momentum value, no worry about cycling the quads. To bring momentum setting up, you need to cycle the quads Q2 and Q3 of either arm. That is increasing the current of Left Q2/Q3 to 1600 A or 1400 for Right arm, wait for 2 minutes after they reach max, then tuning down the value to the desired setting. More detailed instructions are on the white board of the counting house. There's no need to cycle Q1, Dipole.

For the Right HRS:

• Go to the middle room, and look for the green button labelled "Right Spectrometer Controls VME"; it is at the lower right of the lower box of green reset buttons in one of the middle racks.
• Push the button, holding it for 3-4 seconds.
• Wait a few minutes, then the readout for the magnets should reappear in Hall A Tools.
• Check the state of the magnets; if they are not ON and at the correct currents, then do the same process as explained above for the left HRS.
• The Cavity Monitors stop working or saturate:

[There is beam, but bcmcav2 and/or bcmcav3 are close to zero ADC channels or negative. Or saturating.]

Reset the Cavity Crate:

• This doesn't seem to happen as of Sept 4, 2009, but if it does page Zafar Ahmed, Bob Michaels, or John Musson in that order.
• Using your ops account, ssh to hlasrv
• At a command prompt type: cd /cs/opshome/medm
• Then at the command prompt type: medme &
• Click on "Execute", then go to File...Open...HappexBpm...HxBpmExpert_DN.adl
• You'll also need to open HxBpmExpert_UP.adl
• You should notice that the status box next to "Daughter PLL Locked" is dark orange.
• Go to a WWW browser, type in http://apcpdu28.jlab.org as the address.
• In the blue box on the left side of the screen, click on "Outlets."
• In the "Select Control Action" box, pick "Reboot immediate," then check box 8 and press "Next."
• Username and password can be found on a piece of paper on the wall near the whiteboard.
• Click "Apply."
• Now you just have to wait for the reboot to take place. It will turn the crate off, wait a bit and turn it back on again.
• To make sure the reset was successful, you should check that on the epics control screens, he heartbeat status buttons on both monitors should be blinking and the "Daughter PLL Locked" and the "Mother PLL Lock" status buttons should both be yellow. If they are not, the reset needs to be done again.

Setting Gains for both cavities. Judge from raw ADC if the gains are ok.

• Set BCM Pre Gain to 4.
• Set BCM Post Gain to 0
• Set X Pre Gain to 2 for 3H02 and 1 for 3H03
• Set X Post Gain to 0
• Set Y Pre Gain to 2 for 3H02 and 1 for 3H03.
• Set Y Post Gain to 0.
See the screen shots from Fall 2009


• Pvdis CODA config is hung up, ROC31 not reporting status




• Find the apar@adaql1 terminal, or open one, and kill the runControl by typing

  > kcoda

• If you get a warning that someone else might be using coda, and whether or not you really want to kill coda, type

  > Y

NOTE: Only CAPS Y will work, y wont work!!


• Reboot ROC31s   Easiest way is to power cycle at g0injereboot   (chan 1,2,3 are resp. g0inj, happexinj, and hatsv18. See passwd on paper near whiteboard.)
• After the ROC has rebooted successfully, bring up the runControl by typing,

  > startcoda

• Again, if you are warned of someone else using coda, and whether or not you want to startcoda, type

  > Y

• The runControl gui should appear. On the runControl gui, click

  > Connect

• Click on Configure, and select Pvdis
• After the configuration has been set properly, which can be checked by observing the bottom window in the runControl, click

  > Download

• Wait utill you see Download Succeded message at the bottom window of the runControl gui. If instead, you see ROC31 not reporting status for # sec, then you need to kill the runControl, and reboot ROC 31 again.

  Rebooting ROC31

  • Check the terminals to see if we are logged on to ROC31 already, which we should be.
  • If we are not logged on to ROC31 already, login from any terminal as

    > rlogin g0inj

  • Reboot the ROC31 crate by typing

    > reboot

  • Once the reboot starts, you will be logged out automatically, which is normal. It takes about 30 sec or so, no longer than 1 minute for the crate to reboot.

  PVDIS How To Page

  ---

  If you still can't find a resolution contact:

  whoever is maintaining this page
  Last modified: TBP