Robert Michaels, rom@jlab.org, Jefferson
Lab Hall A, updated July 2007
Want CODA 2.6 ? See DAQ Wiki on 2.6
PURPOSE:
First a couple disclaimers: Since I am not in the DAQ group, you should not ask me for a release of CODA software. Ask coda@jlab.org. Also I will not cover some important features like ET system (Event Transfer) which is used both to insert events and to obtain them online in real time.
INSTALLATION:
Here is the general procedure for getting started in building a new system: 1) First install CODA. 2) Set up a user account with the proper environment. 3) Set up the CODA database. 4) Before doing anything with a VME or FASTBUS crate, you should run a test setup where the ROC (readout controller) runs on the Linux PC generating ``fake'' data. See the appendix below.
If you can do steps 1)-4), it means you have 90 % of what you need for a simple 1-crate DAQ system. The next step is to download the kernel and coda_roc program on the DAQ crate. For a multiple crate system, you need a trigger supervisor. The steps are: 1) First do steps 1)-4) above; 2) Next, make CODA work with one crate and no trigger supervisor. 3) Make CODA work with one crate and a trigger supervisor. 4) Add more crates.
For the software installation on Linux, you will need the tar files for CODA, CMLOG, and the cross compilers used to compile VME and fastbus code. Typically you put them on /usr/local/coda or in some centralized location. You compile them. However, this is already done in hall A, the stuff is on /adaqfs/coda.
BOOTING COMPUTERS :
On adaql2 we have a file /etc/rc.d/rc5.d which is a link to ../init.d/msqld_s, which is a script that starts MSQL at boot up. Now, if you are working on adaq cluster you've just wasted your time reading this paragraph because its already set up for you.
In order for the VME crates to boot up, the linux box needs to permit insecure logins like rlogin and rsh. The procedure has changed over the years and the following might not pertain to your particular Linux setup; but here are some hints: Put an approrpriate file "rlogin" and "rsh" into /etc/xinetd.d/ and make sure you have /usr/sbin/in.rlogind and /usr/sbin/in.rshd. There may also be a "services" GUI that allows you to conveniently turn on rlogin and rsh. Restart the daemon "xinetd". Next, regardless of which flavor of Unix/Linux, the user who is hosting the session (specified user in vxWorks boot parameter) should have a file ~/.rhosts (that is dot.rhosts) which contains the internet name of the VME computer(s) involved in DAQ. Else they cannot boot. Note, the entries in .rhosts should be the fully-network-qualified names like hallavme8.jlab.org (and not just hallavme8).
A convenient way to set up the vxWorks boot parameters is to use the serial interface on the linux box using 'minicom'. Here are some pointers. There are normally 1 or 2 serial ports, like /dev/ttyS0 and /dev/ttyS1. Run a cable from one of them to the VME serial interface (called 'debug' on the power PCs). Login as root and set up minicom, by typing 'minicom -s' and follow the GUI to set up the serial device (e.g. /dev/ttyS0), with 9600 Bps, 8 bits NI, and both hardware and software flow control `no'. Save this as a dfl file (default setup file for minicom) and change permissions on the device file /dev/ttyS0 so that ordinary users can write there. Then, as an ordinary user, type 'minicom' and you will be connected. Obviously, type 'man minicom' for details.
Though its a little beyond the scope of this documentation, I'll go ahead and include an example boot script for vxWorks for VME with some pointers:
boot device : dc (if power PC) processor number : 0 host name : adaqs3-ep (where to get vxWorks kernel) file name : /adaqfs/halla/apar/vxworks/vx2306_DECFD (here is kernel) inet on ethernet (e) : 129.57.164.13:ffffff00 (IP of VME, the 'fffffff00' stuff inet on backplane (b): is the subnet mask) host inet (h) : 129.57.164.45 (IP corresponding to host) gateway inet (g) : user (u) : adev (use this guy's .tcshrc and .rhosts) ftp password (pw) (blank = use rsh): flags (f) : 0x20 target name (tn) : halladaq6 (corresponds to IP of VME) startup script (s) : /adaqfs/halla/apar/vxworks/halladaq6.boot (further boot other (o) : instructions)
ACCOUNT ENVIRONMENT:
# tcshrc initialization file. # # Needed for running CODA on Linux. # The default shell should be tcsh. # # Warning - For VxWorks targets to successfully boot # from this account the .tcshrc file should # NOT print anything to stdout (no 'echo', etc) source /site/env/syscshrc set history=1000 set autologout=0 set ignoreeof set notify set savehist=1000 set noclobber set autolist setenv CODA /adaqfs/coda/2.2 source $CODA/.setup setenv ITK_LIBRARY $CODA/common/lib/itk2.0 setenv MSQL_TCP_HOST adaql2 setenv MSQL_HOME /usr/local/msql_log setenv EXPID parity setenv SESSION par1 setenv CMLOG_CONFIG ~/cmlog/cmlogrc setenv CMLOG_PORT 8208 # CODA Data will show up in this directory setenv CODA_DATA /adaql2/data2/parity setenv AUTOBOOT TRUE set path = ($path . $CODA_BIN /usr/local/gnu-vxworks/bin /adaqfs/halla/CMLOG_2.01/bin/Linux ~apar/bin) setenv VXWORKS_INC /site/vxworks/5.3/ppc/target/h setenv GCC_EXEC_PREFIX /usr/local/gnu-vxworks/lib/gcc-lib/ setenv CMLOG_LIB /adaqfs/coda/CMLOG_2.01/lib/Linux # Dirty trick (hopefully temporary): put libXm.so.2 into ~apar/lib # One needs libXm.so.2 for 'cedit' (GUI to edit database). setenv LD_LIBRARY_PATH $CODA/Linux/lib:/usr/X11R6/lib:/adaqfs/halla/apar/lib:$CMLOG_LIB
EDITING YOUR DATABASE:
Its a good idea to start with 'cedit'. The first time, click on 'File' and make a 'New database' (with the same name as EXPID in you .tcshrc). You probably only need one database, but please keep it different from mine if you work in hall A. Radically different configurations can be different 'sessions' (using the $SESSION environment variable on the Linux and VME hosts). However, most developers only need one session. Within one session the 'Run Types' are what you select when you configure in runcontrol, and are also what you see when select 'New...' or 'Open...' in 'File' of cedit.
Now when you're making a new 'Run Type' (also called 'Configuration'), you pick elements from the far left and drop them in the big frame. I can never remember which is which, but after you try one, if its not what you want you can get rid of it with 'dismiss'. The minimal configuration you need is one 'ROC', one 'EB', one 'ER', and a 'CODA file'. You have to draw a line from each ROC to the EB, and from the ER to the CODA file, but (wierdly) NOT from the EB to the ER. Draw the line by pointing to the darkly colored square, clicking left mouse, dragging the resulting line to the darkly colored square on the next component (on my box the 'darkly colored square' is green).
When you select a component, you must enter its attributes in a pop-up GUI. Name is like 'ROC1', 'ROC2', 'EB1'. Name and Id number must be UNIQUE. Hint: Look at an existing configuration in an existing database and get an idea of what to do. Its a little tricky, like the readout list must have the form "full-path/list.o mystring" where list.o is the compiled CRL, and there must be a "mystring" after it, with a space between list.o and mystring. You must have a string even if it is literally just "mystring". For fastbus and VME there must be a secondary readout list, usually.
You can also tie scripts to the components, e.g. to run when they do a 'download' 'prestart' or other transition. These are scripts that run on the Linux host and which can do anything you like. Picking which component to tie your script to is a matter of figuring how you want your script to be causally connected to the run. E.g. a 'ROC end' script will execute at the end of a run but before a 'EB end' (I think).
When finishing 'cedit', don't forget to 'Save' (in 'File'). Otherwise you've just wasted your time.
The GUI 'dbedit' also has its uses. When you start the GUI, you have to enter the host that's running MSQL (if its not the local host). Then you open the database and config in an obvious way and edit the elements. Example: To make the data file name: Select 'rtype1_option' where 'rtype1' is one of the configurations. In the value field for 'dataFile' type /adaql2/data1/parity/run_%d.dat. Then, data will appear in the directory with %d replaced by the run number.
There are some bugs in 'cedit' and 'dbedit'. Fortunately not the same bugs, hence the usefulness of two codes. If you are good at MYSQL you might not need these tools. Anyway, since it can cost time to figure out how to overcome them, I give my 'unofficial list of known bugs' here, but your mileage may vary. 1) When you 'save' from cedit, it might make the output file 'test.dat' no matter what it was before. You need to get go into 'dbedit' and change the 'dataFile'. 2) If you have TS component(s) (trigger supervisor), the output must be specified as 'NONE' or you won't get any data. That is, unless you really want your TS component to produce data, which is possible. cedit doesn't do this for you, so you must do it in dbedit. 3) Adding scripts in dbedit is virtually impossible. I've seen things like: It won't add more than one script to a component (complains about this), or if I modify a script name it gets copied to all other script names. You must use cedit to add/modify scripts. 4) Its always a good idea to double check your database after saving it. Don't ask why.
For more hints, Bryan Moffit, a former graduate student, has compiled a useful set of notes about database setup: www.jlab.org/~moffit/CODA_DOC/database.html.
OBSCURE DETAILS:
READOUT LISTS:
The CODA 1.4 manual had some explanation of CRL. However the syntax is a bit different from CODA 2.2 and I'm not aware of any CODA 2.2 documentation about this. It remains true that you can imbed C in CRL. A line ending in semicolon is taken as C (instead of CRL) and the preprocessor passes it directly to the compiler. Similarly, code lines that are sandwiched between a double percent (%%), with one pair before the lines and one pair after, are taken as C. Some notes on how to convert CODA 1.4 to 2.2 CRL list is in the appendix.
NOTE: A good resource is the examples in jlabs1:/site/coda/ 2.2/examples/crl. Imitate those, or the ones in adaql1:~apar/crl.
One important but obscure thing to note: Do not forget to set bigendian_out = 0 if you have a Linux host. (see the examples). Specifically you declare 'extern int bigendian_out' at the top of CRL and then set bigendian_out=0; in download. Failure to do this results in... mysterious failure of communications with ROCs that you will never understand.
To compile type 'makelist mylist.crl ppc' for a Power PC single-board VME computer.
RUNNING CODA:
APPENDIX 1: Running CODA on Linux to test CODA installation: To check that you have installed CODA correctly, it's a good idea to make a "Unix ROC" test. This involves running a coda_roc on the Linux workstation as the generator of data. It tests all features of the CODA installation on Linux (database works, environment ok, runcontrol works, etc). Suppose you have a CRL like unix_list.crl from CODA's examples, e.g. find one on adaq (probably in ~a-onl/crl). Modify it so that the trigger routine is producing fake data that you can recognize. Compile it with "makelist unix_list.crl native". Note, the last arg is "native" = Linux compilation. This makes a shared object file unix_list.so which you specify as the readout list in the CODA database.
Run the components interactively the first time, in seperate windows:
et_start
rcServer
coda_er -i -s $SESSION -n ER1 -t ER
coda_eb -i -s $SESSION -n EB1 -t EB
coda_roc -i -s $SESSION -n ROCU -t ROC
runcontrol
This should be enough to take data on the Linux workstation.
APPENDIX 2: Argh ! I cannot boot my VME crate:
The first time you boot in life, there are unfortunately about a hundred reasons why might not be able to boot. First thing is make sure you have a good RS232 cable. E.g. you may need a null modem (transmit and receive swapped). Also make sure the boot parameters are right (most of problems are with that). And if the host computer is on a different subnet from the VME, you need to specify a gateway in the boot parameters.
Some other possible symptoms and solutions:
1) You get to the Loading... and the board just sits there forever
2) You get something like:
Attached TCP/IP interface to geisc0.
Attaching network interface lo0... done.
Loading...
Error loading file: errno = 0x3d.
3) You get something like:
Attached TCP/IP interface to geisc0.
Attaching network interface lo0... done.
Loading... permission denied
If you get 1) then you either have no valid network connection (ie a
bad cable etc...) or you have no network route to get to the boot host
due to firewalls, routers, etc... (Do NOT run firewalls !).
Another reason for 1) I have found is that the host user's .tcshrc
file does an "echo" of something. Do not echo !! and this is really
hard to figure out because the result is "Loading ..." and no info why.
If you get 2) then you have contacted the host but it does not have rsh
enabled on it so the error translates to "Connection Refused". One possible
check is to try rsh from another computer; if you can't rsh in, that's
your problem.
If you get 3) then rsh is enabled but you have some software control
that is not allowing you in, like a bad .rhosts file. There are several
options here to look at. E.g. you need to have "hallavme8.jlab.org" in
.rhosts, and not just "hallavme8".
APPENDIX 3: MOVING FROM CODA 1.4 to 2.2:
Differences between CODA 1.4 and CODA 2.0 CRL codes
===================================================
Some diffs are trivial:
comments lines are changed from ! to #.
constants have to be announced differently
( e.g. "const ADDR_TIR ...")
Replacement for data buffer:
*(*dabufp)++ becomes *rol->dabufp++
tir[1] used instead of tir[0] because of routines
CDOENABLE and CDODISABLE which use tir[1].
There are some other CODA 2.x pecularities, like the way to
link triggers to trigger routine, the done routine, etc.
You need this kind of thing in CODA 2.x prestart
init trig source VME
link async trig source VME 1 to bob_trig and bob_done
event type 1 then read VME 1
(Don't do the intConnect to (FUNCPTR)trigger which was
done in CODA 1.4. Unfortunately the above assumes that
you are using a CODA standard TIR module, and if you have
you're own interrupt module like the eP energy aparatus,
life is much more complicated for CODA 2.2 )
Here are a few examples of differences (ala 'diff' command).
The "<" are CODA 2.2 and the ">" are the CODA 1.4
< const ADDR_TIR = hex efff0AD0 # base addr of interrupt board
---
> ADDR_TIR = hex efff0AD0 ! base addr of interrupt board
< # bits: 0 - send a signal to release the veto on the gate
---
> ! bits: 0 - send a signal to release the veto on the gate
< tir[1]->tir_csr = 0x80;
---
> tir[0]->tir_csr = 0x80;
< CDODISABLE(VME,1,0);
---
> tir[0]->tir_csr = 0x80;
In CODA 2.2 you need routines like bob_trig and
bob_done which were linked to interrupt (see above).
Here is the minimal routine for doing a readout,
replete with CODA black magic
Remember, don't try to 'printf' inside this.
begin trigger bob_trig
rol->dabufp = (long *)0;
open event type EVTYPE of BT_UI4
*rol->dabufp++ = whatever_data;
close event
end trigger
Hint: See also the examples in jlabs1:/site/coda/
2.2/examples/crl or the ones in adaql1:~apar/crl.
Robert Michaels --
e-mail:
rom@jlab.org