Notes   on   How   to   Install   and   Run   CODA

Robert Michaels,   rom@jlab.org,   Jefferson Lab Hall A,   updated July 2007

This file:     hallaweb.jlab.org/equipment/daq/coda_hints.html

• PURPOSE:    

This note pertains to installation, account setup, and running of CODA 2.x on RedHat Linux. To some degree the notes also apply to SunOS. Further documentation about CODA 2.x is maintained on the   JLab DAQ Group WWW Page. The present page gives some practical tips which supplement the DAQ group's information.

First a couple disclaimers:   Since I am not in the DAQ group, you should not ask me for a release of CODA software. Ask coda@jlab.org. Also I will not cover some important features like ET system (Event Transfer) which is used both to insert events and to obtain them online in real time.

• INSTALLATION:    

  Here is the general procedure for getting started in building a new system: 1) First install CODA.   2) Set up a user account with the proper environment.   3) Set up the CODA database.   4) Before doing anything with a VME or FASTBUS crate, you should run a test setup where the ROC (readout controller) runs on the Linux PC generating ``fake'' data. See the appendix below.

  If you can do steps 1)-4), it means you have 90 % of what you need for a simple 1-crate DAQ system. The next step is to download the kernel and coda_roc program on the DAQ crate. For a multiple crate system, you need a trigger supervisor. The steps are:   1) First do steps 1)-4) above;   2) Next, make CODA work with one crate and no trigger supervisor.   3) Make CODA work with one crate and a trigger supervisor.   4) Add more crates.

  For the software installation on Linux, you will need the tar files for CODA, CMLOG, and the cross compilers used to compile VME and fastbus code. Typically you put them on /usr/local/coda or in some centralized location. You compile them. However, this is already done in hall A, the stuff is on /adaqfs/coda.

• BOOTING   COMPUTERS :    

One Linux PC must run the MSQL daemon which serves the CODA database. In adaq cluster this runs on adaql2. One can run CODA components like runcontrol, or database editors, remotely on other cpu's, so long as the environment is set correctly (e.g. $MSQL_TCP_HOST = adaql2).

On adaql2 we have a file /etc/rc.d/rc5.d which is a link to ../init.d/msqld_s, which is a script that starts MSQL at boot up. Now, if you are working on adaq cluster you've just wasted your time reading this paragraph because its already set up for you.

In order for the VME crates to boot up, the linux box needs to permit insecure logins like rlogin and rsh. The procedure has changed over the years and the following might not pertain to your particular Linux setup; but here are some hints: Put an approrpriate file "rlogin" and "rsh" into /etc/xinetd.d/ and make sure you have /usr/sbin/in.rlogind and /usr/sbin/in.rshd. There may also be a "services" GUI that allows you to conveniently turn on rlogin and rsh. Restart the daemon "xinetd". Next, regardless of which flavor of Unix/Linux, the user who is hosting the session (specified user in vxWorks boot parameter) should have a file ~/.rhosts (that is dot.rhosts) which contains the internet name of the VME computer(s) involved in DAQ. Else they cannot boot. Note, the entries in .rhosts should be the fully-network-qualified names like hallavme8.jlab.org (and not just hallavme8).

A convenient way to set up the vxWorks boot parameters is to use the serial interface on the linux box using 'minicom'. Here are some pointers. There are normally 1 or 2 serial ports, like /dev/ttyS0 and /dev/ttyS1. Run a cable from one of them to the VME serial interface (called 'debug' on the power PCs). Login as root and set up minicom, by typing 'minicom -s' and follow the GUI to set up the serial device (e.g. /dev/ttyS0), with 9600 Bps, 8 bits NI, and both hardware and software flow control `no'. Save this as a dfl file (default setup file for minicom) and change permissions on the device file /dev/ttyS0 so that ordinary users can write there. Then, as an ordinary user, type 'minicom' and you will be connected. Obviously, type 'man minicom' for details.

Though its a little beyond the scope of this documentation, I'll go ahead and include an example boot script for vxWorks for VME with some pointers:

boot device          : dc                    (if power PC)
processor number     : 0 
host name            : adaqs3-ep             (where to get vxWorks kernel)
file name            : /adaqfs/halla/apar/vxworks/vx2306_DECFD    (here is kernel)
inet on ethernet (e) : 129.57.164.13:ffffff00  (IP of VME, the 'fffffff00' stuff 
inet on backplane (b):                          is the subnet mask)
host inet (h)        : 129.57.164.45         (IP corresponding to host)
gateway inet (g)     : 
user (u)             : adev                  (use this guy's .tcshrc and .rhosts)
ftp password (pw) (blank = use rsh): 
flags (f)            : 0x20      
target name (tn)     : halladaq6             (corresponds to IP of VME)
startup script (s)   : /adaqfs/halla/apar/vxworks/halladaq6.boot   (further boot
other (o)            :                                              instructions)

• ACCOUNT   ENVIRONMENT:    

One needs an account that runs tcsh (and good luck running any other shell). This sets up the environment variables and the path so that you can compile readout lists, run CODA, etc. Below is the .tcshrc file (note the dot) used by the account 'apar' on adaql1,l2. An alternative to this is to execute 'source dosetupcoda' using the script dosetupcoda which normally is available with the CODA distribution.

# tcshrc initialization file. 
#
# Needed for running CODA on Linux.
# The default shell should be tcsh.
#
# Warning - For VxWorks targets to successfully boot
#           from this account the .tcshrc file should
#           NOT print anything to stdout (no 'echo', etc)

source /site/env/syscshrc

set history=1000
set autologout=0
set ignoreeof
set notify
set savehist=1000
set noclobber
set autolist
 
setenv CODA /adaqfs/coda/2.2
source $CODA/.setup
setenv ITK_LIBRARY $CODA/common/lib/itk2.0

setenv MSQL_TCP_HOST adaql2
setenv MSQL_HOME /usr/local/msql_log

setenv EXPID parity
setenv SESSION par1

setenv CMLOG_CONFIG ~/cmlog/cmlogrc
setenv CMLOG_PORT 8208

# CODA Data will show up in this directory
setenv CODA_DATA /adaql2/data2/parity

setenv AUTOBOOT TRUE

set path = ($path . $CODA_BIN /usr/local/gnu-vxworks/bin /adaqfs/halla/CMLOG_2.01/bin/Linux ~apar/bin)

setenv VXWORKS_INC /site/vxworks/5.3/ppc/target/h 

setenv GCC_EXEC_PREFIX /usr/local/gnu-vxworks/lib/gcc-lib/

setenv CMLOG_LIB /adaqfs/coda/CMLOG_2.01/lib/Linux

# Dirty trick (hopefully temporary): put libXm.so.2 into ~apar/lib
# One needs libXm.so.2 for 'cedit' (GUI to edit database).

setenv LD_LIBRARY_PATH $CODA/Linux/lib:/usr/X11R6/lib:/adaqfs/halla/apar/lib:$CMLOG_LIB

• EDITING   YOUR   DATABASE:    

There are two GUI's which edit the MSQL database. They are 'cedit' and 'dbedit'. When I first wrote this there was unfortunately no good documentation about these GUI's, though maybe that's changed in the meantime. Anyway, here are a few hints, which is a admittedly a poor substitute for a user's manual.

Its a good idea to start with 'cedit'. The first time, click on 'File' and make a 'New database' (with the same name as EXPID in you .tcshrc). You probably only need one database, but please keep it different from mine if you work in hall A. Radically different configurations can be different 'sessions' (using the $SESSION environment variable on the Linux and VME hosts). However, most developers only need one session. Within one session the 'Run Types' are what you select when you configure in runcontrol, and are also what you see when select 'New...' or 'Open...' in 'File' of cedit.

Now when you're making a new 'Run Type' (also called 'Configuration'), you pick elements from the far left and drop them in the big frame. I can never remember which is which, but after you try one, if its not what you want you can get rid of it with 'dismiss'. The minimal configuration you need is one 'ROC', one 'EB', one 'ER', and a 'CODA file'. You have to draw a line from each ROC to the EB, and from the ER to the CODA file, but (wierdly) NOT from the EB to the ER. Draw the line by pointing to the darkly colored square, clicking left mouse, dragging the resulting line to the darkly colored square on the next component (on my box the 'darkly colored square' is green).

When you select a component, you must enter its attributes in a pop-up GUI. Name is like 'ROC1', 'ROC2', 'EB1'. Name and Id number must be UNIQUE. Hint: Look at an existing configuration in an existing database and get an idea of what to do. Its a little tricky, like the readout list must have the form "full-path/list.o mystring" where list.o is the compiled CRL, and there must be a "mystring" after it, with a space between list.o and mystring. You must have a string even if it is literally just "mystring". For fastbus and VME there must be a secondary readout list, usually.

You can also tie scripts to the components, e.g. to run when they do a 'download' 'prestart' or other transition. These are scripts that run on the Linux host and which can do anything you like. Picking which component to tie your script to is a matter of figuring how you want your script to be causally connected to the run. E.g. a 'ROC end' script will execute at the end of a run but before a 'EB end' (I think).

When finishing 'cedit', don't forget to 'Save' (in 'File'). Otherwise you've just wasted your time.

The GUI 'dbedit' also has its uses. When you start the GUI, you have to enter the host that's running MSQL (if its not the local host). Then you open the database and config in an obvious way and edit the elements. Example: To make the data file name: Select 'rtype1_option' where 'rtype1' is one of the configurations. In the value field for 'dataFile' type /adaql2/data1/parity/run_%d.dat. Then, data will appear in the directory with %d replaced by the run number.

There are some bugs in 'cedit' and 'dbedit'. Fortunately not the same bugs, hence the usefulness of two codes. If you are good at MYSQL you might not need these tools. Anyway, since it can cost time to figure out how to overcome them, I give my 'unofficial list of known bugs' here, but your mileage may vary.   1) When you 'save' from cedit, it might make the output file 'test.dat' no matter what it was before. You need to get go into 'dbedit' and change the 'dataFile'.   2) If you have TS component(s) (trigger supervisor), the output must be specified as 'NONE' or you won't get any data. That is, unless you really want your TS component to produce data, which is possible.   cedit doesn't do this for you, so you must do it in dbedit.   3) Adding scripts in dbedit is virtually impossible. I've seen things like: It won't add more than one script to a component (complains about this), or if I modify a script name it gets copied to all other script names. You must use cedit to add/modify scripts.   4) Its always a good idea to double check your database after saving it. Don't ask why.

For more hints, Bryan Moffit, a former graduate student, has compiled a useful set of notes about database setup:   www.jlab.org/~moffit/CODA_DOC/database.html.

• OBSCURE   DETAILS:    

Here is a random collection of obscure details about the installation which one might never figure out.  
• 1) If when you type ``makelist my.crl ppc'' you get an error like ``as: uncrecognized option'' this means you must put the gnu-vxworks version of ``as'' in front of the Linux version in your path.  
• 2) The database must have a sessions table with the uid and gid of the user. Very obscure errors if this is not set up.  
• 3) The ``hostname'' on the computer must NOT be the fully qualified name including domain, just the short name. (E.g. ``adaql2'' is ok but ``adaql2.jlab.org'' is wrong, and leads to strange error messages.) Now I've forgotten how to fix this, it was a problem with RH7.2.
• 4) But the ``hostname'' must be the fully qualified one for .rhosts, i.e. hallavme8.jlab.org is good, but hallavme8 is bad.
• 5) gnu vxworks issues: I usually put the gnu_vxworks stuff somewhere on a user account. Suppose it is ~me/gnuvxworks. Then you must make 2 links in /usr/local, they are   "ln -s ~me/gnuvxworks gnu-ppc"   and   "ln -s ~me/gnuvxworks gnu-vxworks".
• 6) This is a fun one. You must have the environment variable LD_ASSUME_KERNEL set to 2.4.2 (i.e. setenv LD_ASSUME_KERNEL 2.4.2) for RHEL 3WS and probably 4WS, otherwise coda_er will dump core without explanation. Oh, but for kernel 2.6* this is not necessary and is even harmful (as I just recently learned).

• READOUT   LISTS:    

The readout lists are written in a special language called CRL (CODA Readout List). This is a preprocessor which converts your CRL into C. Its convenient because it takes care of inserting headers and standard blocks of code. If you have a list like 'mylist.crl' the preprocessor makes 'mylist.c' (C code) which then gets compiled to 'mylist.o', which then gets downloaded to the crate.

The CODA 1.4 manual had some explanation of CRL. However the syntax is a bit different from CODA 2.2 and I'm not aware of any CODA 2.2 documentation about this. It remains true that you can imbed C in CRL. A line ending in semicolon is taken as C (instead of CRL) and the preprocessor passes it directly to the compiler. Similarly, code lines that are sandwiched between a double percent (%%), with one pair before the lines and one pair after, are taken as C.   Some notes on how to convert CODA 1.4 to 2.2 CRL list is in the appendix.

NOTE:   A good resource is the examples in jlabs1:/site/coda/ 2.2/examples/crl.   Imitate those, or the ones in adaql1:~apar/crl.

One important but obscure thing to note:   Do not forget to set bigendian_out = 0 if you have a Linux host.   (see the examples). Specifically you declare 'extern int bigendian_out' at the top of CRL and then set bigendian_out=0; in download. Failure to do this results in... mysterious failure of communications with ROCs that you will never understand.

To compile type 'makelist mylist.crl ppc' for a Power PC single-board VME computer.

• RUNNING   CODA:    

Running 'runcontrol' and taking data is straight forward. I refer you to the guides for Hall A shift workers:  hallaweb.jlab.org/equipment/daq/guide.html.   and    hallaweb.jlab.org/equipment/daq/guide2.html.

• APPENDIX 1:     Running CODA on Linux to test CODA installation: To check that you have installed CODA correctly, it's a good idea to make a "Unix ROC" test. This involves running a coda_roc on the Linux workstation as the generator of data. It tests all features of the CODA installation on Linux (database works, environment ok, runcontrol works, etc). Suppose you have a CRL like unix_list.crl from CODA's examples, e.g. find one on adaq (probably in ~a-onl/crl). Modify it so that the trigger routine is producing fake data that you can recognize. Compile it with "makelist unix_list.crl native". Note, the last arg is "native" = Linux compilation. This makes a shared object file unix_list.so which you specify as the readout list in the CODA database.

  Run the components interactively the first time, in seperate windows:
  
            et_start
  
            rcServer 
  
            coda_er -i -s $SESSION -n ER1 -t ER
  
            coda_eb -i -s $SESSION -n EB1 -t EB
  
            coda_roc -i -s $SESSION -n ROCU -t ROC
  
            runcontrol
  
  This should be enough to take data on the Linux workstation.
  
  

  APPENDIX 2:     Argh !   I cannot boot my VME crate:    
  The first time you boot in life, there are unfortunately about a hundred reasons why might not be able to boot. First thing is make sure you have a good RS232 cable. E.g. you may need a null modem (transmit and receive swapped). Also make sure the boot parameters are right (most of problems are with that). And if the host computer is on a different subnet from the VME, you need to specify a gateway in the boot parameters.

  Some other possible symptoms and solutions:
  
    1)  You get to the Loading... and the board just sits there forever
  
    2)  You get something like:
           Attached TCP/IP interface to geisc0.
           Attaching network interface lo0... done.
           Loading...
           Error loading file: errno = 0x3d.
  
    3)   You get something like:
           Attached TCP/IP interface to geisc0.
           Attaching network interface lo0... done.
           Loading... permission denied
  
     If you get 1) then you either have no valid network connection (ie a
  bad cable etc...) or you have no network route to get to the boot host
  due to firewalls, routers, etc...  (Do NOT run firewalls !).
  
     If you get 2) then you have contacted the host but it does not have rsh
  enabled on it so the error translates to "Connection Refused".  One possible
  check is to try rsh from another computer; if you can't rsh in, that's
  your problem.
  
     If you get 3) then rsh is enabled but you have some software control
  that is not allowing you in, like a bad .rhosts file. There are several
  options here to look at.  E.g. you need to have "hallavme8.jlab.org" in 
  .rhosts, and not just "hallavme8".
  
  

  APPENDIX 3:     MOVING FROM CODA 1.4 to 2.2:    
This is probably obsolete since I guess nobody uses 1.4.   Gee, I can't wait for CODA 3.0.

 Differences between CODA 1.4 and CODA 2.0 CRL codes
 ===================================================

  Some diffs are trivial:  
     comments lines are changed from ! to #.
     constants have to be announced differently 
        ( e.g. "const ADDR_TIR ...")
  Replacement for data buffer:
      *(*dabufp)++  becomes *rol->dabufp++

  tir[1] used instead of tir[0] because of routines 
  CDOENABLE and CDODISABLE which use tir[1].

  There are some other CODA 2.x pecularities, like the way to 
  link triggers to trigger routine, the done routine, etc.
  You need this kind of thing in CODA 2.x prestart
    init trig source VME
    link async trig source VME 1 to bob_trig and bob_done
    event type 1 then read VME 1

  (Don't do the intConnect to (FUNCPTR)trigger which was
   done in CODA 1.4.  Unfortunately the above assumes that
   you are using a CODA standard TIR module, and if you have
   you're own interrupt module like the eP energy aparatus,
   life is much more complicated for CODA 2.2 )

  Here are a few examples of differences (ala 'diff' command).  
  The "<" are CODA 2.2 and the ">" are the CODA 1.4

< const ADDR_TIR  = hex efff0AD0    # base addr of interrupt board
---
> ADDR_TIR = hex efff0AD0    ! base addr of interrupt board


< # bits:  0 - send a signal to release the veto on the gate
---
> ! bits:  0 - send a signal to release the veto on the gate 


<   tir[1]->tir_csr = 0x80;
---
>   tir[0]->tir_csr = 0x80;


<    CDODISABLE(VME,1,0);
---
>    tir[0]->tir_csr = 0x80; 


In CODA 2.2 you need routines like bob_trig and 
bob_done which were linked to interrupt (see above).

Here is the minimal routine for doing a readout,
replete with CODA black magic
Remember, don't try to 'printf' inside this.

begin trigger bob_trig 
  rol->dabufp = (long *)0;
  open event type EVTYPE of BT_UI4
  *rol->dabufp++ = whatever_data;
  close event
end trigger

Hint:  See also the examples in jlabs1:/site/coda/
2.2/examples/crl or the ones in adaql1:~apar/crl.

Robert Michaels   --   e-mail: rom@jlab.org