Difference between revisions of "DVCS how to"
(→Reboot calorimeter VME ADCs crate)
|Line 411:||Line 411:|
Login on the src2 host ( use the username and password for hacweb8 ) and go outlet control and select reboot immediate on the selected plug.
Login on the src2 host ( use the username and password for hacweb8 ) and go outlet control and select reboot immediate on the selected plug.
Here is a detailed procedure [[http://www.jlab.org/~adaq/halog/html/1011_archive/101121081830.html]]
Here is a detailed procedure [[http://www.jlab.org/~adaq/halog/html/1011_archive/101121081830.html]]
Revision as of 14:37, 1 December 2010
Back to DVCS
(This page is obviously in progress -- please contribute to it)
- 1 ioc_portserver_reboot_locations
- 2 Troubleshooting
- 3 Shift-worker responsibilities
- 4 Beam Recovery Procedure
- 5 Data Acquisition
- 5.1 Basic Data Acquisition with CODA
- 5.2 Triggers, Scalers, and Dead Time
- 5.3 Cold Start Checklist (when beam has been off for several hours)
- 5.4 Starting CODA From Scratch
- 5.5 How to Take a Pedestal Run
- 6 Data analysis
- 7 Spectrometer detectors
- 7.1 How to change the momentum setting of HRS
- 7.2 How to change the spectrometer angle
- 7.3 How to reset the Bogie IOC
- 7.4 How to reset the quadrupoles
- 7.5 How to reboot HRS magnet IOC
- 7.6 How to restart the HRS hall probe gaussmeters
- 7.7 What/where to check VDC
- 7.8 How to set / reset phototube high voltage
- 7.9 How to reset the VDC high voltage
- 7.10 How to start xscaler (HRS) / view the scalers
- 8 Beam Line
- 9 Compton Operation
- 9.1 Running the Compton DAQ
- 9.2 Compton Stripchart
- 9.3 Checking the Offline Analysis
- 9.4 Compton Troubleshooting
- 10 Slow control
- 11 Miscellaneous
- The start and end of run entries do not show up in HALOG:
- Make sure you have waited enough. Automated entries take a while to show up in the HAlog.
- You want to quit coda, connect to adaql2 as a-onl using ssh, anbd restart coda. Yes even if you are already connect on adaql2 as a-onl, the ssh part is what you need here.
- Deadtime reading [this halog entry]
- If the DVCS calorimeter DC readings are frozen: read that []
Every one taking shift should have read, understood and sign the [safety documentation] off the experiments.
- Follow the runplan and communicate with RC or experts whenever problems happen that cannot be solved by shift workers.
- Maintain data taking quality and an efficient use of beam time.
- Follow the directives in the COO
- Log the following in a shift summary:
- run list (describing the goal of this run: eg production on LH2, BMC calibration ...) and report main statistic numbers main stat howto
- all major events including accesses.
- Watch the target, see Target Info
- Since being TO is easy, please help the shift leader and third person.
- Start and stop the DAQ.
- Do online replay of all production runs;
- Compare replay histograms with the sample ones and report to shift leader any unexplained differences. Halog them.
- Fill the shift check list once a shift.
- The shift leader or TO can help.
Beam Recovery Procedure
Update: October 24, 2010.
- Empty target, establish correct beam position
- 5 uA empty target
- BPM4A (X,Y) = ( 0.0, 0.0) mm
- BPM4B (X,Y) = ( 0.0, 0.0) mm
- See white board for updates on these.
- Tolerances on these : +/- 0.3 mm
- In standard configuration we run without the raster (ok on cryotarget if beam current is less than 5uA)
- Put in BeO target, for visual check of spot (if camera still works)
- 10 uA rastered beam on BeO.
Basic Data Acquisition with CODA
This assumes runcontrol is already up. If not, go to Starting CODA From Scratch below.
Use the configs (these may change):
- As we are commissioning the experiment, the configuration to take data on the calo and the LHRS changes a lot. The most current configuration is written on the white board. If it is not, ask the RC.
- For taking data without the calorimeter choose configuration "LeftHRS" (Two Fastbus crates on L-HRS, ROC3 = lower crate, ROC4 = upper crate).
Data appear in /adaql2/data*/dvcs10_NNN.dat.X where NNN is the run number. You can find the last run with the command "lastrun"
Triggers, Scalers, and Dead Time
Setting prescale factor
The prescale factors are in: /adaqfs/home/a-onl/prescale/prescale.dat
The comments at the top tell you which number corresponds to each trigger. Edit the string at the bottom of the file to modify the prescale settings.
Starting xscaler (HRS) and checking trigger rates
- Use hapc5 which is the PC on the top at the far right of counting room console. Normally xscaler will be running, but if not, log on as adaq, type "xscaler" and do what is commanded of you.
- If you need to start xscaler type: xscaler.
How to download and check the trigger
Read Bob Michael's last message about setting the trigger: [2010-10-13's halog entry]
To change the trigger:
- Logon to an adaq Linux box like "adaql1" or "adaql2" as user "atrig".
A self-explanatory graphical user interface pops up and you should do what it says and press "Download" and WAIT for it to finish and do not press Ctrl-C. The user should look for suspicious error messages in the window from which trigsetup was launched, e.g. to check if connection to the crate is ok.
- More details are described in Bob Michaels' page Loading and Checking the Hall A HRS Trigger.
How to check the dead time
Read this [halog] first. If it is not already running, type datamon in a terminal on adaql2 in the a-onl account.
Definitions of DVCS triggers
DVCS+LHRS DAQ on adaql2/a-onl account Prescale factors (integer), downloaded each run at prestart. Change this file only in-between runs. Maxima: (2^24)-1 = 16777215 for T1-T4 and 65535 for T5-T8 PS factor of zero DISABLES the trigger. This means T9 - T12 are disabled unless you have a PS factor >0 (however, T9 - T12 cannot actually be prescaled, only disabled/enabled)
- T1 = L-HRS S2 trigger
- T2 = L-HRS S0 trigger
- T3 = L-HRS S1.and.S2 trigger
- T4 = L-HRS efficiency (2/3 among S1,S2,C)
- T5 = main DVCS coinc trigger (ARS Valid)
- T6 = ARS Stop
- T7 = Calo LED pulser
- T8 = 1024 Hz pulser
- T9 = a copy of TSETTLE
String should be one line, no spaces, ending in carriage return Do NOT put spaces in string, NOR add 2nd string. (REALLY!!!)
Cold Start Checklist (when beam has been off for several hours)
- Confirm target is “empty target”
- Call for pulsed beam for tuning
- Tune beam to beam dump (straight through, no Compton chicane)
- Call for 10μA beam
- establish beam position
- check beam charge monitors
- check beam position monitors
- Confirm HRSs set to correct value
- Move target to THIN Carbon position for detector check out
- For detector checkout: current = 0.5μA
- Check out detectors at different currents
- In standard production configuration, check that the raster is off.
- Resume runplan
Starting CODA From Scratch
You do not need to do this routinely - only if you are starting from a fresh login or after a computer crash. Log in a a-onl account on adaql2 ( check with Run Coordinator in case you don't know the password )
(Contact: Bob Michaels)
On one workspace you will see six xterms with names of the crates in
the title, e.g. "ROC4--hallasfi4--hatsv40-port6".
From each of the six xterms, you should telnet into the appropriate RS232 interface, e.g. for the above, "telnet hatsv40 2006" (if the prompt in the xterm is "
->", it is already connected). The "hallasfi4" is the internet name of the crate's cpu.
If the xterms are missing, type "setupxterms".
From these xterms, you reboot by typing "ctrl-X". Yes, you hold down the Ctrl key, then hit X. You can also type reboot but "ctrl-X" is better. You can also reboot from the crate resets GUI on HA Tools.
To power cycle the DVCS crates, type crateOff in a terminal in logged as a-onl and after crateON to turn both ROC17 and ROC18 back on.
Open a terminal and type startcoda.
The runcontrol window will appear: File:Startcoda.png
Hit the "Connect" button, then "Configure" (choose a configuration e.g. LeftHrs), and "Start Run", or, to troubleshoot, "Download", "Prestart", and "Go".
How to Take a Pedestal Run
If you've been running the normal spectrometer DAQ configuration, you'll need to press "Abort", then "RunTypes" and select the PEDRUN configuration. Download, StartRun. Run for about 5000 events, then EndRun. You may check the pedestal files in ~/ped (pedN.dat corresponding to ROC #N). See README there for more details. After pedestal run, change back to the running configuration.
NOTE: DONT try to use PEDRUN for anything other than pedestal determination -- its confusing since the prescale factors are in a different file, etc. If you want a run with pedestal suppression turned off, see the README.
login to adaql4, adaql5, adaql6 or adaql7 with user adaq
> analyzer onlana.C
Once the analyzer has worked its way through the data, look at the data with the following commands:
- .x online.C+("detectorL", NNNN);
- .x online.C+("calo", NNNN);
- .x online.C+("physics", NNNN);
where NNNN is the run number.
Batch offline analysis
As soon as CODA files go to /mss, a batch job is submitted to the JLab farm to decode each file. Outputs are store under /mss/home/munoz/DVCS2/1stprod. A copy of the last 100 files is kept in the work disks under /work/halla/dvcs/disk1/1stprod.
Starting the Port Server Windows
On adaql2 type : start_monitors
this will open 6 xterms with in the title on which portserver to log :
so for example to log on hatsv4 port 3 type :
telnet hatsv4 2003
How to change the momentum setting of HRS
From the HAC window, there is a P0 SET field for each spectrometer in which you can enter the desired momentum setting. To change to a lower momentum from a higher previous setting, you just need to enter the new momentum value. To raise the momentum setting, you need to cycle the Q2 and Q3 magnets first and then enter the desired momentum setting. To cyle the Q2 and Q3, turn off their regulation (Hall A General Tools, click the button right of P0set), send Q2 and Q3 to 1600 A, wait two minutes at 1600A, set the p0 to the deisred value, turn onn Q2 and Q3 regulations. If you run into trouble with the dipole NMR lock, [this] might help you.
How to change the spectrometer angle
First, do a visual survey of the Hall to make sure nothing is in the spectrometer's path. Also, make sure you are not exceeding the minimum (typically 14 degrees) or maximum (experiment dependent) spectrometer angle.Second, on hacsbc2, click the red "tool box" icon on the linux taskbar, as above. Choose bogies_SetSpec so that you can determine the angle and vernier setting for the spectrometer.
Third, choose bogies_Left (or bogies_Right) in the tool box to bring up the bogies control screen. [image:bogies_RIGHT.window.gif] Click PSM enable and wait a few seconds for PSM OK to read YES. Click DM enable and wait a few seconds for DM OK to read YES.
Make sure the velocity is set to 0 and the direction is CW or CCW as desired. Click on Brake Release and wait for Brakes OK to read YES.
Click on ClampRelease, set the velocity to 700. Once you see the spectrometer start to move in the floor angle camera - you cannot see the spectrometer move in the Hall overview camera, as it only moves a few degrees per minute at maximum speed. For the left arm, to move to a larger angle, the direction should be CCW, while for the right arm CW moves the spectrometer to larger angle. The direction of the spectrometer is reversed by using a negative rpm. Watch the spectrometer motion on the cameras. When you are getting close to the desired angle, slow down to about 300 rpm. To stop, click on the Clamp Release button and the Brake button. Disable DM and PSM, and disconnect to close the GUI. Read off the floor angle mark and vernier, and input the values into the appropriate fields in the Alignment section of the Hall A General Tools GUI.
If the bogie become unresponsive, you might read the section below on how to reset the bogie.
How to reset the Bogie IOC
Sometimes the HRS bogie controllers get in a funny state and will not respond to the GUI. For example, you click on the buttons (PSM enable, DM enable, etc), but the state field doesn't change, even after a patient wait of 5--10 seconds.
The first thing you should try is to close the Bogie GUI window using the disconnect button, wait for 5 minutes and try again. Sometimes the system will correct itself. If it doesn't you may want to try rebooting the Bogie IOC.
Follow this link: Resetting the HRS Bogie IOC
How to reset the quadrupoles
The reset for Q2 is simple, on the front of the power supply there are red lights on when the power supply is tripped. At the bottom of the board that has the red light there is a blue reset button. There may be more than one light on so there may be more than one blue button that needs to be pushed. Push blue buttons until all of the red lights are out. Once all of the lights are out, lower the large lever on the lower right side of the power supply and lift it back into position. this will reset the Q2 and Q3 power supplies locally. For Q1, it's simpler : there is a large sign in the back of the Q1 equipment rack on the power supply balcony that says the blue reset button is here. you press the button and the magnet resets.
How to reboot HRS magnet IOC
Go to the hall A main menu screen.
Click "IOCS". Note that the left HRS magnet IOC's (iocha14) are accessible at http://hareboot10 and the right HRS magnet IOC's(iocha16) are accessible at http://hareboot25 (Username and password for IOC reboot are available in the counting house). Close the "Hall A IOC Systems" window. To reboot:
- the left HRS magnet IOC's, go to http://hareboot10 and from the pull-up menu next to iocha14, select "immediate reboot".
- the right HRS magnet IOC's, go to http://hareboot25, click on control, select iocha16, from the pull-up menu select "immediate reboot".
After the IOC reboot, please make sure that the spectrometer-angle information is reentered into the Hall A General Tools GUI.
How to restart the HRS hall probe gaussmeters
This procedure is NEW as of September 28, 2010.
From HALOG 333032:
The Lakeshore gaussmeters for the magnet hall probes are no longer being controlled by the Slow Controls VME crates (iocha14 and iocha16). You will notice that the GUI for each gaussmeter is slightly different than the previous GUI. The functions are still the same. However, if a meter "hangs", a person must notify accelerator Software On Call to reset the interface; ask MCC to make the call. At present this is not something we can do from the Counting House.
DO NOT attempt to reboot iocha14 or iocha16 to correct any problem with a gaussmeter readout. The control is now done through a portserver with an Accelerator PC acting as a "soft"-IOC. We do not have direct access to this PC.
What/where to check VDC
VDC status can be checked on the HAC Main Control Window. High Voltages and Gas Flows are constantly being monitored.
How to set / reset phototube high voltage
From the Hall A Main Menu on hacsbc2, click on LeCroy HV. Choose the appropriate spectrometer, and then the appropriate detector. Usually the voltages are already plateaued, and you should only need to turn HV on. All voltage numbers are in Volts.
How to reset the VDC high voltage
Bring up the HAC control window per the above instructions. In the "Miscellaneous" section, click on the "Crate Resets" button. This will bring up the "Hall A Slow Controls" window. In the bottom left and right of this window are the reset buttons for the spectrometer subsystems. Click on the relevant "VDC high voltage" reset button. You then need to go back to the main HAC GUI and reset the tripped VDC to 4 kV by entering 4 in the VDC HV window. If the VDC trips again, reset the supply then try to ramp up the voltage slowly. If there are repeated trips, call an expert.
How to start xscaler (HRS) / view the scalers
start xscaler :
- Go to the hapc5 terminal in the middle of rack CH01A09. If it is not logged in, log in as user "adaq"
- Type ssh adaq@adaql4 and enter the password when prompted.
- Type goxscaler
- Type ./xscaler
to view the scalers :
- The simplest : login to adaq l1 as adaq
- Type xscaler
- Follow what it says
NB : you must be in the correct directory since there are 2 versions of xscaler : the new is ROOT GUI (the previous one is Calvin Howell's GUI)
- Make sure that the 12V box on the screen "HRS hadron systems FPP carbon doors" are always on (HallA mainmenu, FPP button, carbon doors).
How to Check the Raster (spot)
- Log onto "adaql2" as user "adaq"
- Type spot++ from any directory
- After checking the spot for a run make changes with MCC according to the needed spot size (note: MCC units are NOT in mm).
Running the Compton DAQ
- To start the compton DAQ, log onto compton@compton and type:
> coda reboot
- The correct CODA configuration is FADC_Prod
- To see the ROC window, open another terminal on compton@compton and type:
> telnet hatsv5 2005
- Start runs with the Start Run button, and stop them with the End Run button. Compton runs do not end automatically, and should last about one hour.
- At the end of the run, a grey box will appear. Type a short description of the run into the box (i.e. "production") and the hit the Submit button.
- It is important to monitor the Compton data online with a stripchart.
- To open the Compton stripchart, open a terminal on compton@compton and type:
> StripTool HappexIIICompton.stp
- The stripchart should look like:
- Purple line: Beam current (uA). In the figure you can see that it has tripped a few times.
- Yellow line: Vertical beam position. This should remain roughly constant, although it fluctuates during dithering cycles. A beam trip sends it to the bottom of the chart (position 0).
- Red line: Cavity power. This should be a square wave: the cavity is locked in the right polarization state, then turns off, then locks in the left polarization state, then turns off, then repeats. The powers for the right and left states appear different but are actually the same. The power should stay comfortably above 2kW. (Cavity may not lock during extended beam trips.) If the cavity is not locking, contact a Compton expert.
- Green line: Current in the photon detector PMT base. This should remain very constant during stable running. It will go to zero (the top of the chart) if the HV trips off. When this happens the HV needs to be turned back on.
- Pink line: Scattered photon rates not normalized to beam current.
- Blue line: Scattered photon rates normalized to beam current in the detector. As in the figure, these should track the cavity on/off states; if you can tell from the blue line alone whether the cavity is on or off, then you are probably getting good Compton data. If not, see the Compton Troubleshooting section
Checking the Offline Analysis
- The offline analysis takes about 45 minutes to run and begins after you hit the submit button.
- The analysis results appear in the Compton Logbook. The run number is on the far left, and the link to the html page brings up the analyzed plots.
- If the data is good production data, the plots should look somewhat like this.
The Compton rate (the blue line on the stripchart) flatlines or goes to zero;
This is a problem because it means that we are not getting good data
- Is there beam? If the beam has tripped, the rate should be zero and there is no problem.
- Is there a run going? If the run has ended the rate goes to zero. Start a new run.
- Has the high voltage tripped? If the green line on the stripchart has gone to zero, the high voltage needs to be turned back on.
- The Compton high voltage is in the beamline crate, card 11, channel 1. If this channel is at zero, and channels 7 and 8 are both on, channel 1 needs to be reset. This is done by turning the channel off and then back on again.
- If the entire crate has tripped off (all three channels in card 11) use the HV/OFF and then HV/ON buttons at the top of the HV GUI.
- If the high volgate needs to be reset make a HALOG entry, including if it was the crate or the channel which tripped off.
- If it's tripping over and over again, contact a Compton expert.
- Is the DAQ running?
- Sometimes when CODA crashes the photon rate flatlines. This means the rate does not change at all for quite some time (although the normalized rate will change during beam trips even if the DAQ has crashed).
- CODA needs to be rebooted. You must end the run submit the run. Then open a terminal on compton@compton and type coda reboot. To run CODA, hit the connect button, then the configure button. Choose the FADC_Prod configuration, download, and then you can go back to running like normal.
CODA is giving a ROC 6 error;
- If CODA is giving a ROC 6 error and the ROC 6 window is not responding, the ROC needs to be rebooted
- This is done by going to hareboot4 in a web browser. The username is hlauser and the password is written on the wall of the counting house. Go to outlet 6 and choose Immediate Reboot and then click the apply button on the bottom of the screen. Wait a few minutes until the ROC is rebooted and then re-download the FADC_Prod configuration in CODA.
The laser (the red line on the stripchart) is always zero;
This is a big problem because it means that there is no laser to Compton scatter off of, and we are therefore getting no data.
- This requires an expert; call a Compton expert.
If something is wrong, and you don't know how to fix it, contact a Compton expert! Bold text
How to bring up HAC control window (Hall A Tools)
The machine which runs the HAC GUI is hacsbc2. The monitor is located in the Hall A Counting House in rack CH01A02, just below the NMR dipole oscilloscopes. Click here to see the screen.
Open up a terminal (make sure you are logged in as hacuser, see the paper at the upper right corner of the whiteboard for password) and type NewTools at the prompt. This will open a window with one button: "EOS menu". Left-click on the button and select "EDM (HLA Main)". A separate window entitle "Hall A Main Menu" will appear. One of the entries is labeld "Tools". Clicking on the "Tools" button opens the "Hall A General Tools" screen, which is often referred to as the HAC window.
High Voltage controls (including DVCS calo)
[We do not use the right arm]: For the right arm, login in an adaq computer as adev, and launch the program hvgui_vnc.
For the left arm and beamline, go in the Lecroy item in Hall A EPICS menu
For the DVCS calorimeter:
The HV control should be already running and you should be able to interact with it through a VNC server.
- connect as adev on any adaql machine,
- type "hvgui" to start the vnc window.
- at the top of the HV gui, select "maps" then DVCS for a human friendly display
In case the HV control is not running, you need to restart it through the VNC server wrapper:
- connect as adev on adaql10,
- type "start_hvgui".
For more information about the HV control for the DVCS calorimeter see HV_CALO
LED table control
The LED driver is located in the DVCS IOC. To access it:
- open a terminal, and type the command: "telnet iochacs"
- load the LED control library by typing: "ld < NewLedLib.o"
Then it is possible to control the led. Few commands:
- SetFrequency(prescalefactor1,prescalefactor2), to turn off the LED put a large prescale factor SetFrequency(0xffff,0xffff)
- To switch ON the LED corresponding to a particular block: SetBlock(columnNbr, rowNbr)
- To scan several blocks of the calo with the LEDs: ScanCalo(MaxColumnNbr, MaxRowNumber, Number)
- To turn the Led Off LedOff()
- To turn the Led On LedOn()
Reboot calorimeter VME ADCs crate
If the DC current are not read out ( ie GUI frozen and time out in the end of run ) that could mean that the IOC iochacs is down.
The IOC is plugged on a remoter control power strip which is using the hostname src2
The plug assignements are
5 DVCS Bottom HV
8 network switch
12 DVCS Top HV
End CODA run first (see modified instructions []) Login on the src2 host ( use the username and password for hacweb8 ) and go outlet control and select reboot immediate on the selected plug. Here is a detailed procedure []
If no halog guis are up and running, log in as user "adaq" on any "adaq.." machine. Type halog a yellow and blue and gray window will appear Enter text in the large message area, your name in the User area, and a title in the Keywords area. Remember that experiments tend to suffer from too little documentation, not too much. Also, halog entries allow people to remotely check on the experiment progress without bothering you with phone calls. When complete, after pressing the "Make entry" button, the text in the message center to the lower right corner of the window should read 'Entry Complete', otherwise something is wrong. It often takes several minutes for entries to show up in halog. Entering graphics: Click 'Grab Screen' in the bottom button row of the halog window. The halog window will disappear, and the window the cursor is in will become the active window, its outline flashing. By moving the cursor you can select any window to be selected, by clicking on the left mouse button. You also can select an arbitrary part of the screen by holding down the left mouse button and dragging the cursor along. All the time a rectangle will flash that indicates the selected area. Releasing the mouse button will then cause this screen area to be included into the entry. It will show up as a thumbnail picture in the halog window, where you can discard it or accept it, and then make a second screen grab. The images are in GIF format.
How to bring up the Alarm Handler
From the "toolbox" icon in the taskbar, press the "Alarm Handler" label. Alarms are indicated by a beep and the alarm handler button flashing. The color red indicates values far out of range, while the color white indicates a communication failure. Clicking on the alarm handler button will bring up an alarm handler list. A detailed list is given in the left panel, and clicking on items here brings up a more detailed list in the right panel. The alarm can be acknowledged and silenced by clicking on the button beside the alarm indicator.
You can plot time dependencies of EPICS variables
Pressing the "toolbox" icon and launch "StripTool", which enables you to plot any EPICS variable(s) versus time. To find out the name of a particular EPICS variable, click on it with the middle mouse button on the MEDM GUI.
How to bring up the Beamtime Time Accounting Table
- Logon to "adaql1" as user "adaq"
- type "bta"
See http://hallaweb.jlab.org/equipment/daq/abu.html for more information.
What/where to check target temperature and pressure
The target temperature and pressure is displayed on the target GUIs running on cryotarg as well as on the camera monitors in the rack (CH01A06) left of the target computer. Look at the camera monitor on the top right of rack CH01A06. It displays the temperatures for the three loops on three white temperature controllers, with loop 1 at the top and loop 3 on the bottom. Ask the TO for help if you can't find the temperature or it doesn't make sense. The adjacent camera displays the JT valve controller readings and tachometers.