Running   CODA   on   Spectrometers   and   Parity   DAQ   Systems

Robert Michaels, rom@jlab.org, Jefferson Lab Hall A, updated Jan 2003

This file: hallaweb.jlab.org/equipment/daq/guide2.html

I.  Running Spectrometer and Parity DAQ

This assumes that runcontrol is running.   If not, see Cold Start in section III.

• Three DAQ Systems
  • Use the "adev" account on adaqs2 for Left Spectrometer.
  • Use the "a-onl" account on adaql2 for Right Spectrometer.
  • Use the "apar" account on adaql1 for Parity DAQ.
  • The run coordinator should know the passwords.   For details on the attributes of these systems see:   hallaweb.jlab.org/equipment/daq/guide.html.
• Normally, runcontrol will be already running. To start and stop runs, push the buttons "Start Run" and "End Run".   Start Run is the same as the sequence prestart, go.
• In order to "synchronize" the runs, start and stop runs for the two spectrometers and Parity DAQ within a minute of each other. Unlike previously, there is no hardware synchronization at the moment.
• To change configurations, use the "Run Type" button. If you've been running, you will first have to push "Abort" button. Choose the configuration from the dialog box, then press "Download". The relevant configurations for 2-Trigger-Supervisor modes are:
  • LeftUnbuff -- (unbuffered only) to run the Left Spectrometer on adaqs2.
  • RightHRS -- (buffered mode) to run the Right Spectrometer on adaql2.
  • gdh -- to run the Parity DAQ on adaql1.

  • Each spectrometer DAQ has a pedestal run configuration -- pedrunL, pedrunR. A pulser run generates the pedestal files for pedestal suppression. A 5 minute run is sufficient. (Note: PEDRUN uses a different prescale file, the data goes to disk6 or 7, the datamon window doesn't know about it. Only use this config for getting pedestals, see section on pedestal runs below.) Gory details are at adaqs2:~adev/ped/README or adaql2:~a-onl/ped/README.

II.  Frequently Asked Questions

• Where is the data ?     To find run 1602, type "find_run 1602" from adaq Sun computers. Another command: "lastrun" prints where the latest run is. The Left Arm data are stored on disks /adaqs2/dataN (N=1,2,3...), the Right Arm data are stored on /adaql2/dataN (N=1,2,3...), and the Parity data are stored on /adaql1/data1/parity as files parity03_M.dat where N=1,2,3... and M = run number.
• What is the deadtime ?   The deadtime is displayed in the "datamon" window -- make sure you are using the one on same computer as runcontrol. Parity DAQ has no "datamon" (and no deadtime). Probably datamon is already running near the runcontrol window. If it is not running, type "datamon"
• Prescale Factor ?   Go to ~/prescale and edit file "prescale.dat" where   ~   is ~adev or ~a-onl
• Disk Handling ?   There are at least 300 Gbytes of disk on adaqs2 and adaql2. Automatic scripts choose the disk to write data, copy to MSS, delete the data when its safe, and avoid simultaneous read/write on a disk. Things to *NOT* do:
  • Do *NOT* erase or move data files (even if you think its junk).
  • Do *NOT* try to change run numbers.
  • Do *NOT* change names of run files.
  • Do *NOT* try to put data by hand into MSS.

  It takes a couple hours for files to appear in MSS, and the files remain on adaq disk for typically 2 or 3 days.

• Pedestal Runs If you've been running the normal spectrometer DAQ configuration, you'll need to press "Abort", then "RunTypes" and select the pedrunL or pedrunR configuration. Download, StartRun. Run for about 5000 events, then EndRun. You may check the pedestal files in ~/ped (ped1.dat, ped2.dat, ped3.dat), where ~ is adev or a-onl, as appropriate. See README files there for details. After pedestal run, change back to the running configuration. NOTE: DONT try to use PEDRUN for anything other than pedestal determination -- its confusing since the prescale factors are in a different file, the data goes to disk 6, etc. If you want a run with pedestal suppression turned off, see the section in the README.

HOW TO SHUTDOWN or REBOOT WORKSTATIONS

Rarely, the workstations don't function properly and the simplest way out is to reboot. To reboot adaqs2,s3 (SunOS): Login as "adaq" and type "reboot". In a few minutes the workstation comes back alive. This solves most problems. Shutdown is necessary in case of power failure. To shutdown, login as "adaq" and type "shutdown". After several minutes screen goes black, wait a minute more, then power off. When you power on again, computer reboots. For Linux, Ole usually keeps some instructions posted near the PC terminal.   One may hit Ctrl-Alt-F1 to go to console mode, then Ctrl-Alt-Del.

QUICK   RESETS

Problems with CODA can usually be solved with a simple reset or with a Cold Start. If not call Bob Michaels. Also can call Bodo Reitz for Spectrometer DAQ and Bryan Moffit for Parity DAQ. Do NOT waste an hour stuck on resets. I claim I can bring CODA up in less than 10 minutes 90% of the time.

If a ROC seems to be hung up, you can reboot it by going to the workspace "Components" and typing "reboot" at the vxWorks prompt (-> reboot). Wait 2 minutes and telnet back in to verify its alive. The name of the ROC computer is normally written in the name of the xterm window. You need to know what subset of these computers are used for your configuration. For GDH experiment, the following are used:   Left Arm uses 1) ROC3-hallasfi3-(hatsv4, port2);   2) ROC4-hallasfi4-(hatsv4, port3);   3) ROC14-hallavme1-(hatsv5, port14),   4) TS1-hallavme4-(hatsv4, port 14). Notation means for example ROC3 is the CODA component, hallasfi3 is the network name (hallasfi3.jlab.org), and the RS232 connection is on portserver hatsv4.jlab.org at port 2. Continuing, the RightHRS uses:   1) ROC1-hallasfi1-(hatsv3, port 2)   2) ROC2-hallasfi2-(hatsv3, port 8);   TS0-hallavme2-(hatsv3, port 3).   The Parity DAQ uses one crate:   ROC23-halladaq6-(hatsv9, 2004). If a ROC seems really frozen, use the "Crate Resets" button in the General Tools EPICS screen on hacsbc. Here the notation is roc14=bpm/raster, roc1 and roc2 are tied together to SFI#1 (R-arm) roc3 and roc4 are tied to SFI#1 (L-arm). TS0=R-arm Trig Supervisor. TS1=L-arm Trig Supervisor. This GUI does not have the Parity crate, but that crate is located in the middle room of the counting house (see the big sign) and can be power cycled if necessary. To reset from the GUI, toggle the state of the button. If you reboot the ROC, or if something on the workstation is hung up, try pushing the "Reset" button in runcontrol. Then Configure, Download, and StartRun as usual.

I emphasize that the procedure of just booting the one crashed ROC and Resetting runcontrol is the quickest solution to most crashes, and it is not necessary to reboot or reset a different DAQ, as they are decoupled. If this reset procedure doesn't work, try a Cold Start (see below).

COLD   START

First kill all CODA process on the workstation where CODA is running by typing "kcoda" from the relevant account and computer. This stops runcontrol, the event builder, event recorder, the runcontrol server, and cleans up the ET system. Now you can start everything again -- look at the printout from "kcoda" script which tells you what else to do, which typically involves starting "rcServer" and "coda_er" by hand, plus resetting runcontrol after connect and before download. Note: rcServer, coda_er, coda_eb must be left in the foreground of a window.

Before downloading, it is first a good idea to make sure the fastbus and VME crates (i.e. the ROCs) are running. One finds that when resetting, one can frequently avoid rebooting the frontend crates, and just restart runcontrol, but if you must: Press the reset buttons for the relevant subset of crates as explained in the Reset section above. A convenient way to reset (reboot) the ROCs is to go to the Components workspace and enter "reboot" at the vxWorks prompt -> of each ROC. But one can also use the EPICS GUI as explained above. After a few minutes, telnet back in and verify they are up. Some of the crates take longer than others to reboot. Be patient. To start everything on the workstation, do what was printed out by "kcoda". As it explains, start "rcServer" first. Then start "coda_er" and "coda_eb" interactively with the arguments as "kcoda" explained. These can also start automatically by CODA but running them interactively provides useful diagnostics (e.g. if coda_eb reports "event type mismatch" it is a sign of severe loss of synch). Finally, type "runcontrol". In runcontrol press the Connect button. After "connect", wait 10 seconds, then press "Run Type"; a dialog box pops up and you must choose the configuration you want, which is normally the first choice. Then press download and wait about 30 seconds. If download hangs up, press "Cancel", "Reset", and "Download" in runcontrol. After a successful download, you can "Start Run" to start a run.

After bringing up runcontrol from a crash its a good idea to "Reset" after configuring the "Run Type". If its ever necessary to reprogram the NVRAMS on the ROCs, see adaqs2:/home/adev/doc/vmeram.doc. Sometimes connection via portserver (RS232 interface) is useful, see adaqs2:/home/adaq/doc/portserver.doc

If you ever logout of the DAQ computer (e.g. adaqs2), or have rebooted it, here is how to restart the preferred setup. We'll run our two spectrometer DAQs and parity DAQ from the adaqs2 terminal. Login there as adev. Passwords are written on a piece of paper on the wall. Run runcontrol on the workspaces called "LeftHRS", "RightHRS", and "Parity" for the 3 DAQs. Login to the respective accounts (adev account on adaqs2 is L-arm; you're already there. a-onl on adaql2 for R-arm, and apar on adaql1 for Parity). In the HRS workspaces, start emacs in background: emacs ~/prescale/ prescale.dat &     Also start datamon by typing "datamon". In the "components" workspace, login to all the frontend computers by typing "telnet hallasfi1" where the name of the computer like hallasfi1 is in the name of the xterm window. If these xterm windows for components aren't there, type "setupxterms".

MSQL server: When you reboot adaqs2 you must start msql by hand by executing the following command from adev account:   /etc/rc3.d/S99coda   I don't know why the system doesn't start it for you.

• Lots of deadtime -- Please see hallaweb.jlab.org/equipment/daq/dtime_faq.html
• Computer/Network problems: Try calling Bob Michaels (pager 584-7410). You may also call the computer-center expert on-call (cell phone 876-1794). For problems with the linux boxes or ESPACE ask Ole Hansen, though Bob can probably help too. For problems with slow controls or hac ask Javier Gomez. Dave Abbott (7190) is Bob's substitute when out of town. Bryan Moffit is an expert on Parity DAQ.

• The Event Recorder (ER) or Event Builder (EB) may complain something like "ER1 not responding" and you get no events. Solution: before running "runcontrol" one must run "rcServer" interactively from the login shell of the relevant account on the relevant computer.

• MSQL not running on adaqs2 (Sun): If you do 'ps -ef | grep msqld' and don't see msqld running under adev, then type the following as adev account '/etc/rc3.d/S99coda'

• CODA may get stuck downloading. This has been happening lately with Left HRS DAQ. Solution: Press "Cancel" in top left corner of runcontrol, then "Reset", and then "Download" again.

• How to Program the RAM of the ROCs. When fastbus loses power, it may be necessary to reload the VME RAM (it isn't as non-volatile as it should be.) Up to date info is kept at adaqs2:/home/adev/doc/vmeram.doc
• If Bob is unavailable, some details about recovery are at jlabs1:/home/rom/doc/recovery.doc

This page maintained by Robert Michaels   rom@jlab.org