Running   CODA   on   Spectrometers   DAQ   Systems

Robert Michaels, rom@jlab.org, Jefferson Lab Hall A, updated June 2004

This file: hallaweb.jlab.org/equipment/daq/guide2.html     Warning:   This is NOT for HAPPEX DAQ, which is a seperate DAQ.

I.   NORMAL   DAQ   OPERATIONS

This assumes that runcontrol is running.   If not, see Cold Start in section III.

• Use the "a-onl" account on adaql2.

• The run coordinator should know the passwords, which are also posted on the wall on a paper near the whiteboard.

• Normally, runcontrol will be already running.   If not, see section on Cold Start below.   To start and stop runs, push the buttons "Start Run" and "End Run".   Start Run is the same as the sequence prestart, go.

• To change configurations, use the "Run Type" button. If you've been running, you will first have to push "Abort" button. Choose the configuration from the dialog box, then press "Download". The relevant configurations are:

• TWOARM -- Reads both spectrometers but NO RICH.

• PEDRUN -- a pedestal run for Fastbus only

II.  FREQUENTLY   ASKED   QUESTIONS

• Where is the data ?     To find run 1602, type "find_run 1602" from adaq computers on the adaq accounts. Another command: "lastrun" prints where the latest run is. The data are on /adaql2/dataN (N=1,2,3,..)
• What is the deadtime ?   The deadtime is displayed in the "datamon" window -- make sure you are using the one on same computer as runcontrol. Probably datamon is already running near the runcontrol window. If it is not running, type "datamon"
• Prescale Factor ?   Go to ~a-onl/prescale and edit file "prescale.dat"
• Disk Handling ?   There are several hundred Gbytes of disk for data taking. Automatic scripts choose the disk to write data, copy to MSS, delete the data when its safe, and avoid simultaneous read/write on a disk. Things to *NOT* do:

• Do *NOT* erase or move data files (even if you think its junk).
• Do *NOT* try to change run numbers.
• Do *NOT* change names of run files.
• Do *NOT* try to put data by hand into MSS.

• Pedestal Runs   The purpose of a pedestal run is to obtain pedestal files used by fastbus for pedestal suppression.   If you've been running the normal spectrometer DAQ configuration, you'll need to press "Abort", then "RunTypes" and select the PEDRUN configuration. Download, StartRun. Run for about 5000 events, then EndRun. You may check the pedestal files in ~a-onl/ped (pedN.dat corresponding to ROC #N). See README there for more details. After pedestal run, change back to the running configuration. NOTE: DONT try to use PEDRUN for anything other than pedestal determination -- its confusing since the prescale factors are in a different file, etc. If you want a run with pedestal suppression turned off, see the README.

HOW TO SHUTDOWN or REBOOT WORKSTATIONS

Rarely, the workstations don't function properly and the simplest way out is to reboot. To reboot adaqs2,s3 (SunOS): Login as "adaq" and type "reboot". In a few minutes the workstation comes back alive.   For Linux, Ole usually keeps some instructions posted near the PC terminal.   One may hit Ctrl-Alt-F1 to go to console mode, then Ctrl-Alt-Del.

QUICK   RESETS

Problems with CODA 2.5 can usually be solved with a simple reset or with a Cold Start. If not call Bob Michaels or Bodo Reitz. Do NOT waste an hour stuck on resets.

If a ROC seems to be hung up, you can reboot it by going to the workspace "Components" and typing "reboot" at the vxWorks prompt (-> reboot). Wait 2 minutes and telnet back in to verify its alive. The name of the ROC computer is normally written in the name of the xterm (hallasfi1, hallasfi2, etc). You need to know what subset of these computers are used for your configuration.   For example, for HAPPEX we use hallasfi1 (R-arm fastbus ROC1), hallasfi2 (R-arm fastbus ROC2), hallasfi3 (L-arm fastbus ROC3), hallasfi4 (L-arm fastbus ROC4), hallavme2 (R-arm Scaler Crate TS0), hallavme4 (L-arm Scaler Crate TS1). The RICH crates are ROC12 and ROC16. If the ROC seems really frozen, use the "Crate Resets" button in the magnet EPICS screen on hac. Some labelling in this GUI: roc14=bpm/raster, roc1=R-arm fastbus1 (this also boots roc2), roc3=L-arm fastbus1, roc4=L-arm fastbus2, TS0=R-arm Trig Super, TS1=L-arm Trig Super. Note, both fastbbus crates in on an arm are reset with one button.   To reset from the GUI, toggle the state of the button. Note, ROC12 and ROC16 cannot be reset via the GUI, they can only be reset by typing "reboot" at the vxWorks prompt.

If you reboot the ROC, or if something on the workstation is hung up, try pushing the "Reset" button in runcontrol. Then Configure, Download, and StartRun as usual.

If a ``quick'' reset doesn't work, try a Cold Start (see below).

COLD   START

First kill all CODA process on the workstation where CODA is running by typing "kcoda" from the relevant account and computer (this should be a-onl account on adaql2 during HAPPEX experiment). "kcoda" stops runcontrol, the event builder, event recorder, the runcontrol server, and cleans up the ET system. Now you can start everything again -- look at the printout from "kcoda" script which tells you what else to do, which typically involves starting "rcServer" and "coda_er" by hand, plus resetting runcontrol after connect and before download. This is probably a matter of arrowing up in the window where the command was last run (the window may have a name like "rcServer", etc). Note: rcServer, coda_er, coda_eb must be left in the foreground of a window.

Before downloading, it is first a good idea to make sure the fastbus and VME crates (i.e. the ROCs) are running. One finds that when resetting, one can frequently avoid rebooting the frontend crates, and just restart runcontrol, but if you must: Press the reset buttons for the relevant subset of crates as explained in the Reset section above. A convenient way to reset (reboot) the ROCs is to go to the Components workspace and enter "reboot" at the vxWorks prompt -> of each ROC. But one can also use the EPICS GUI as explained above. After a few minutes, telnet back in and verify they are up. Some of the crates take longer than others to reboot. Be patient. To start everything on the workstation, do what was printed out by "kcoda". As it explains, start "rcServer" first. Then start "coda_er" and "coda_eb" interactively with the arguments as "kcoda" explained. Finally, type "runcontrol". In runcontrol press the Connect button. After "connect", wait 10 seconds, then press "Run Type"; a dialog box pops up and you must choose the configuration you want, which is normally the first choice. Then press download and wait about 30 seconds. If download hangs up, press "Cancel", "Reset", and "Download" in runcontrol. After a successful download, you can "Start Run" to start a run.

After bringing up runcontrol from a crash its a good idea to "Reset" after configuring the "Run Type".

If you ever logout of the DAQ computer adaql2 or have rebooted it, here is how to restart the preferred setup. Login as "a-onl". Emacs the prescale file in the background: "emacs ~a-onl/prescale/prescale.dat &"   Also start datamon by typing "datamon".   Next, in the "components" workspace, login to all the frontend computers by typing "telnet hallasfi1" where the name of the computer like hallasfi1 is in the name of the xterm window. Also in the name of these xterms is the portserver port where you can connect via RS232 (portserver instructions are at hallaweb.jlab.org/equipment/daq/portserver.html). So, an example name may be an xterm with title "ROC2--hallasfi2--hatsv3-port-8" meaning ROC2 is IP address hallasfi2 and on portserver hatsv3 at port 8.   Note: If these xterm windows for components are not there, type "setupxterms" to bring them up.

• Lots of deadtime -- Please see hallaweb.jlab.org/equipment/daq/dtime_faq.html
• Loss of network connectivity in the hall. Symptoms: Cannot run DAQ, cannot ping any VME computer, cannot ping the laptops on the detectors stack (ahut1, ahut2). cannot ping the network switch in the hall (129.57.192.2). Solution: Enter the hall, find the 192 switch in the 2nd or 3rd rack from entrance to hall, along the beamline (hopefully labelled; if not, look for little yellow cables with labels like "ROC4", "TS1", "moller", etc) and power cycle it.
• Computer/Network problems: Try calling Bob Michaels (pager 584-7410). You may also call the computer-center expert on-call (cell phone 876-1794). For problems with the linux boxes or ROOT analyzer ask Ole Hansen, though Bob can probably help too. For problems with slow controls or hac ask Javier Gomez. Bodo Reitz and Dave Abbott are Bob's substitute when out of town.

• The Event Recorder (ER) or Event Builder (EB) may complain something like "ER1 not responding" on the Linux version of CODA, and you get no events. This may be because you forgot: Before running "runcontrol" one must run "rcServer" interactively from the login shell of the relevant account on the relevant computer ("a-onl" on "adaql2").

• How to Program the RAM of the ROCs. When fastbus loses power, it may be necessary to reload the VME RAM (it isn't as non-volatile as it should be.) Up to date info is kept at ~adev/doc/vmeram.doc   Portserver instructions ~adaq/doc/portserver.doc which are also online.
• If Bob is unavailable, some details about recovery are at jlabs1:/home/rom/doc/recovery.doc

More info: hallaweb.jlab.org/equipment/daq/guide.html   and   hallaweb.jlab.org/equipment/daq/daq_trig.html.