Running   CODA   on   Spectrometers   and   Parity   DAQ   Systems

Robert Michaels, rom@jlab.org, Jefferson Lab Hall A, updated July 2001

This file: hallaweb.jlab.org/equipment/daq/guide2.html

I.  Running Spectrometer and Parity DAQ

This assumes that runcontrol is running.   If not, see Cold Start in section III.

• Three DAQ Systems
  • Use the "adev" account on adaqs2 for Left Spectrometer.
  • Use the "a-onl" account on adaql2 for Right Spectrometer.
  • Use the "apar" account on adaqcp for Parity DAQ.
  • The run coordinator should know the passwords.   For details on the attributes of these systems see:   hallaweb.jlab.org/equipment/daq/guide.html.
• Normally, runcontrol will be already running. To start and stop runs, push the buttons "Start Run" and "End Run".   Start Run is the same as the sequence prestart, go.
• In order to "synchronize" the runs, StartRun for the two spectrometers first (in any order), then StartRun the Parity DAQ last. EndRun Parity first, then the two spectrometers. For more info about synchronization, see section IV below.
• To change configurations, use the "Run Type" button. If you've been running, you will first have to push "Abort" button. Choose the configuration from the dialog box, then press "Download". The relevant configurations for 2-Trigger-Supervisor modes are:
  • LeftSpect -- to run the Left Spectrometer on adaqs2.
  • RightSpect -- to run the Right Spectrometer on adaql2.
  • happex1 -- to run the Parity DAQ on adaqcp.

  • Each spectrometer DAQ has a pedestal run configuration -- pedrunL, pedrunR. A pulser run generates the pedestal files for pedestal suppression. A 5 minute run is sufficient. (Note: PEDRUN uses a different prescale file, the data goes to disk6 or 7, the datamon window doesn't know about it. Only use this config for getting pedestals, see section on pedestal runs below.) Gory details are at adaqs2:~adev/ped/README or adaql2:~a-onl/ped/README.

II.  Frequently Asked Questions

• Where is the data ?     To find run 1602, type "find_run 1602" from adaq computers on the adaq accounts. Another command: "lastrun" prints where the latest run is. The Left Arm data are stored on disks /adaqs2/dataN (N=1,2,3...), the Right Arm data are stored on /adaql2/dataN (N=1,2,3...), and the Parity data are stored on /adaql2/dataN/parity as files parity01_M.dat where N=1,2,3... and M = run number.
• What is the deadtime ?   The deadtime is displayed in the "datamon" window -- make sure you are using the one on same computer as runcontrol. Parity DAQ has no "datamon" (and no deadtime). Probably datamon is already running near the runcontrol window. If it is not running, type "datamon"
• Prescale Factor ?   Go to ~/prescale and edit file "prescale.dat" where   ~   is ~adev or ~a-onl
• Disk Handling ?   There are at least 300 Gbytes of disk on adaqs2 and adaql2. Automatic scripts choose the disk to write data, copy to MSS, delete the data when its safe, and avoid simultaneous read/write on a disk. Things to *NOT* do:
  • Do *NOT* erase or move data files (even if you think its junk).
  • Do *NOT* try to change run numbers.
  • Do *NOT* change names of run files.
  • Do *NOT* try to put data by hand into MSS.

  It takes a couple hours for files to appear in MSS, and the files remain on adaq disk for typically 2 or 3 days.

• Pedestal Runs If you've been running the normal spectrometer DAQ configuration, you'll need to press "Abort", then "RunTypes" and select the pedrunL or pedrunR configuration. Download, StartRun. Run for about 5000 events, then EndRun. You may check the pedestal files in ~/ped (ped1.dat and ped2.dat), where ~ is adev or a-onl, as appropriate. See README files there for details. After pedestal run, change back to the running configuration. NOTE: DONT try to use PEDRUN for anything other than pedestal determination -- its confusing since the prescale factors are in a different file, the data goes to disk 6, etc. If you want a run with pedestal suppression turned off, see the section in the ~adev/ped/README or ~a-onl/ped/README. For Parity DAQ there is no concept of a pedestal run.

HOW TO SHUTDOWN or REBOOT WORKSTATIONS

Rarely, the workstations don't function properly and the simplest way out is to reboot. To reboot adaqs2,s3 (SunOS): Login as "adaq" and type "reboot". In a few minutes the workstation comes back alive. This solves most problems. For adaqh2 (HP-UX), you have to "shutdown" (cannot "reboot"). Shutdown works on SunOS and HP-UX, and is necessary in case of power failure. To shutdown, login as "adaq" and type "shutdown". After several minutes screen goes black, wait a minute more, then power off. When you power on again, computer reboots. (Eccentricity of adaqs2: When it comes back on from shutdown, there is a sign "Ok" where you have type "boot". When you login to online account you have to type "xhost +adaqs2".) For Linux, Ole usually keeps some instructions posted near the PC terminal.   One may hit Ctrl-Alt-F1 to go to console mode, then Ctrl-Alt-Del.

QUICK   RESETS

Problems with CODA 2.2 can usually be solved with a simple reset or with a Cold Start. If not call Bob Michaels. Also can call Bodo Reitz for Spectrometer DAQ and Antonin Vacheret for Parity DAQ. Do NOT waste an hour stuck on resets. I claim I can bring CODA up in less than 10 minutes 90% of the time.

If a ROC seems to be hung up, you can reboot it by going to the workspace "Components" and typing "reboot" at the vxWorks prompt (-> reboot). Wait 2 minutes and telnet back in to verify its alive. The name of the ROC computer is normally written in the name of the xterm (hallasfi1, hallasfi2, etc). You need to know what subset of these computers are used for your configuration. For example, in summer 2001 we use use hallasfi1 (ROC1), hallavme2(ROC15), and halladaq1(TS0) for Right Spectrometer, hallasfi2(ROC2), hallavme1(ROC14), and halladaq4(TS1) for Left Spectrometer, and halladaq6(ROC1) for Parity DAQ. If the ROC seems really frozen, use the "Crate Resets" button in the magnet EPICS screen on hac. roc14=bpm/raster, roc1=R-arm fastbus1, roc2=L-arm fastbus1, roc3=L-arm fastbus2, TS0=R-arm Trig Super, TS1=L-arm Trig Super. This GUI doesn't have the Parity crate, but that crate is located in the middle room of the counting house (see the big sign) and can be power cycled if necessary. To reset from the GUI, toggle the state of the button. If you reboot the ROC, or if something on the workstation is hung up, try pushing the "Reset" button in runcontrol. Then Configure, Download, and StartRun as usual.

I emphasize that this procedure of just booting the one crashed ROC and Resetting runcontrol is the quickest solution to most crashes, and it is not necessary to reboot or reset a different DAQ, as they are decoupled. If this reset procedure doesn't work, try a Cold Start (see below).

COLD   START

First kill all CODA process on the workstation where CODA is running by typing from anywhere on the relevant account "kcoda". This stops runcontrol, the event builder, event recorder, the runcontrol server, and cleans up the ET system. Now you can start everything again by typing "runcontrol" (but NOTE: for Linux versions like R-arm and Parity DAQ, one should start "rcServer" interactively before starting "runcontrol". This avoids some problems like "ER not connected". Just type "rcServer" and you can iconize that window.)

Before downloading, it is first a good idea to make sure the fastbus and VME crates (i.e. the ROCs) are running. One finds that when resetting, one can frequently avoid rebooting the frontend crates, and just restart runcontrol, but if you must: Press the reset buttons for the relevant subset of ROC1, ROC2, ROC14, ROC15, TS0, TS1 etc that belong to the DAQ you are resetting, as explained in the Reset section above. A convenient way to reset (reboot) the ROCs is to go to the Components workspace and enter "reboot" at the vxWorks prompt -> of each ROC. After a few minutes, telnet back in and verify they are up. Some of the crates take longer than others to reboot. Be pateient. If the ROCs are frozen, press the reset buttons instead of typing "reboot". To start everything on the workstation, type "runcontrol". Then press the Connect button. After "connect", wait 10 seconds, then press "Run Type"; a dialog box pops up and you must choose the configuration you want, which is presumably "twospect" for two-arm operation. Then press download and wait about 30 seconds. Now you can "Start Run" to start a run.

After bringing up runcontrol from a crash its a good idea to "Reset" after configuring the "Run Type". If its ever necessary to reprogram the NVRAMS on the ROCs, see adaqs2:/home/adev/doc/vmeram.doc. Sometimes connection via portserver (RS232 interface) is useful, see adaqs2:/home/adaq/doc/portserver.doc

If you ever logout of the DAQ computer (e.g. adaqs2), or have rebooted it, here is how to restart the preferred setup. We'll run our two spectrometer DAQs and parity DAQ from the adaqs2 terminal. Login there as adev. Run runcontrol on the workspaces called "LeftSpect", "RightSpect", and "Parity" for the 3 DAQs. In these workspaces, start emacs in background: emacs ~/prescale/prescale.dat &     Also start datamon by typing "datamon". In the "components" workspace, login to all the frontend computers by typing "telnet hallasfi1" where the name of the computer like hallasfi1 is in the name of the xterm window. If these xterm windows for components aren't there, type "setupxterms". For details on the attributes of the 3 independent DAQ systems see:   hallaweb.jlab.org/equipment/daq/guide.html.

A hardware level is sent from the Parity DAQ to the two Spectrometer DAQs in order to synchronize them, i.e. to make the run duration identical. This was seen as desirable in order to make the runs measure the same running conditions. When starting a run, the experimenters should first start the two spectrometer DAQs (in any order), and last start the Parity DAQ. When ending a run, the Parity DAQ should be ended first and the spectrometer DAQs ended subsequently.

One subtlety is that if both spectrometers fail to end a run, the scaler gates will be off (until the next run). To clear the inhibit, one may type ``clearinb'' from the nearly any account (e.g. adaq) on Linux or SunOS. Another way to clear the inhibit is to simply remove the cable sent from Parity DAQ. The cable is labeled near the Parity DAQ crate, located in the middle room of the counting room. Removing this inhibit makes the 3 DAQ systems independent and has no other implications, bad or otherwise.

• Lots of deadtime -- Please see hallaweb.jlab.org/equipment/daq/dtime_faq.html
• Computer/Network problems: Try calling Bob Michaels (pager 881-7897). You may also call the computer-center expert on-call (cell phone 876-1794). For problems with the linux boxes or ESPACE ask Ole Hansen, though Bob can probably help too. For problems with slow controls or hac ask Javier Gomez. Dave Abbott (7190) is Bob's substitute when out of town.

• The Event Recorder (ER) or Event Builder (EB) may complain something like "ER1 not responding" on the Linux version of CODA, and you get no events. The workaround was recently explained : before running "runcontrol" one must run "rcServer" interactively from the login shell of the relevant account on the relevant computer (e.g. a-onl account on adaql2).

• How to Program the RAM of the ROCs. When fastbus loses power, it may be necessary to reload the VME RAM (it isn't as non-volatile as it should be.) Up to date info is kept at adaqs2:/home/adev/doc/vmeram.doc
• If Bob is unavailable, some details about recovery are at jlabs1:/home/rom/doc/recovery.doc

This page maintained by Robert Michaels   rom@jlab.org