HALL  A  RAW  DATA  STRUCTURE   --   YEAR 2001

R. Michaels,  Jefferson Lab,  e-mail:  rom@jlab.org,  updated May 14, 2001

•  OVERVIEW

The raw data structure recorded in the spectrometer DAQ datastream will be described.  This includes the following data:  1) Event data recorded with each accepted trigger  (fastbus and VME data from the two spectrometers and from the BPM and raster);  2) scalers;  3)  EPICS information;  and 4)  "special events" which are inserted once or infrequently during a run, such as prescale information and detector map.

Fortunately, there is already a large body of software which decodes and interprets these data, so in many cases the experimenter may use higher level variables, such as presented in ESPACE kumacs.  In addition to ESPACE, there are some auxiliary routines (from L. Todor, for example) which interpret the scalers and EPICS data.

I will not describe the various other datastreams in Hall A, such as Moller polarimeter,  e-P energy measurement,  Compton Polarimeter, etc.

A note about naming scheme. Prior to Sept 2000 the detector package which is now on the Right Spectrometer was called the Electron Arm. The package on the L-arm was the H-arm. In Sept 2000 the packages were swapped with a crane operation and we realized the old naming scheme made no sense (e.g. both spectrometers can have electrons). The old naming still exists in some places like detector map or some software, though we should be slowly weeding it out. Also, it is quite relevant for older experiments.

To find out about the scalers prior to the year 2001, see the URL:   dstruct_pre2001.html.

•  CODA  DATA  FORMAT

The best detailed reference on the CODA format was the old manual for CODA version 1.4, see appendix E.   I have saved this postscript file coda1.4.ps.   I will give a simplified summary here.  The data come in the form of an array of 32-bit words, in the case of event data, or character data, in the case of some of the special events.  In general, the  first part of the event structure is "header" information, indicating how long the event is, what the event type is, etc.  The most important header and other identifying data words are:

Identifying CODA Data Words

Location in Buffer	Content	Description
1st  word	event length	loop from J=1 to J=evbuffer(1)+1 in Fortran to access data = evbuffer(J)
ishft( evbuffer(2), -16 )	event  type	Types 1-12 are triggers that don't overlap in time.   Type 14 is any trigger overlap.  Other types correspond to prestart, go, end, etc. Types >128 are "special events".
evbuffer(4) in event type 17	run number	The run number, which is normally in the filename also.
evbuffer(5)	event number	The event number, uniquely assigned by CODA

After the header, the data are organized in banks corresponding to each ROC.   A ROC is a "readout controller", and it means a crate of fastbus or VME.   We have four ROCs in our spectrometer DAQ event stream:  1) ROC1 = fastbus on E-arm (after 9/2000 it is called R-arm);  2) ROC2 = fastbus on H-arm (L-arm) for VDCs and scintillators; 3) ROC3 = fastbus on H-arm for FPP;  4) ROC14 = VME crate for BPM and raster data. In some periods there were more ROCs (13, 15).  The beginning of each ROC bank contains the length of the bank and an identifying tag to indicate which ROC number.  A bit of Fortran code which illustrates how to crawl through the banks is shown below.

          do 150 i = 1,MAXROC
c n1 = pointer to first word of ROC
            if(i.eq.1) then
               n1 = evbuffer(3) + 4
            else
               n1 = n1roc(irn(i-1))+lenroc(irn(i-1))+1
            endif
c irn = ROC number obtained from CODA
            irn(i) = ishft(iand(evbuffer(n1+1),'1f0000'x),-16)
            if(irn(i).le.0.or.irn(i).gt.maxroc) then
               write(6,*) 'fatal error in decoding'
               stop
            endif
c Store pointers and lengths
            if(i.eq.1) then
               n1roc(irn(1)) = n1
               lenroc(irn(1)) = evbuffer(n1)
               lentot = n1 + evbuffer(n1)
            else
               n1roc(irn(i)) = n1roc(irn(i-1))+lenroc(irn(i-1))+1
               lenroc(irn(i)) = evbuffer(n1roc(irn(i)))
               lentot = lentot + lenroc(irn(i)) + 1
            endif
            if(lentot.ge.len-1) goto 200
150     end do
200     numroc = i

Libraries exist for reading and writing CODA data format. The library is in the ESPACE distribution, in the /evio directory. A convenient example code for reading/writing CODA data (written in C) is here: rwcoda.c with a Makefile (one can link to CODA libraries or simply compile evio.c and link it instead).

If you want to be modern and hip, try my   CODA classes   (C++ classes for CODA interface).

•   DECODING  OF  DATA

The data are decoded by ESPACE in the routine event_decoder which is in the module coda.f.   This routine requires the detector map.  A recent example of the detector map is shown here: detmap.config.   This detector map gives the correspondence between the "raw channel number" and the "analysis detector channel".  By "raw channel number" I mean the combination of :  (ROC number, slot in ROC,  channel within the slot).  By "analysis detector channel"  I mean the channel within a detector as indexed by ESPACE.  The detector map is also stored in the CODA file as a special event and can be used by ESPACE by leaving out (or commenting out)  the line "set/file/detmap detmap.config" in the ESPACE kumac.  The absence of this line causes ESPACE to search the CODA file for the detmap.  The presence of this line causes ESPACE to take the detmap specified in the kumac, which then over-rides whatever is written in the CODA file.  The intention, of course, is that the detmap in the CODA file is the correct one at the time of the experiment.  There have been some reports that ESPACE occassionally (rarely) fails to find the detmap in a CODA file.  You can work around this problem by extracting it once and then renaming this file and using it as the the file pointed to in the kumac file.  The extracted detmap file is always called detector_map.lis.

To add a new detector to ESPACE, there is a recipe which affects several routines.  To learn the recipe, ask me.  If the new detector is added this way, it allows for incorporation in the detector map.

•   SCALERS

Scalers count raw hits on phototubes, as well as  important quantities like charge and triggers which are used to normalize the experiment.  The scalers are read online using the "xscaler" code, and the meaning of the channels is evident from the scaler configuration file which that code uses.  The scaler config file is normally located in the directory from which you must run "xscaler", namely ~adaq/$EXPERIMENT/right/scaler ~adaq/$EXPERIMENT/left/scaler where $EXPERIMENT is an environment variable like e94104.  

We attempt to keep the channel mapping the same, but some changes are inevitable. NOTE:   As of May 2001, the info about scalers is up-to-date, and is backward compatible up to Jan 1, while old information about running prior to 2001 has been into   dstruct_pre2001.html (see section about scalers). The information here is only valid for the year 2001.

The number of scaler channels in a bank is determined by the lowest 6 bits of the header. Some scaler have 16, others 32 channels.

Arrangement of  the Right   Spectrometer   Updated May 14, 2001

Bank of 16 or 32 Channels	Header	Usage
1st  Bank	0xceb00020	1-6=S1-Left, 7,8=blank, 9-13 and 15 =S2-Left, 14,16=blank, 17-22=S1-Right, 23,24=blank, 25-29 and 31 =S2-Right, 30,32=blank.
2nd  Bank	0xceb20010	Left.AND.Right PMTs: 1-6=S1, 7,8=blank, 9-13 and 15 =S2, 14,16=blank.
3rd  Bank	0xceb30010	1-10=Gas Cerenkov PMTs, 11=GasC-sum
4th  Bank	0xceb50010	EDT Scaler after May 14, 2001. (under development)
5th  Bank	0xceb60010	Spare channels
6th  Bank	0xceb70020	Plus Helicity gated normalization data. Channels 1-32 follow the same map as channels 1-32 of the ceb8 scaler. Prior to May 1, 2001, chan 17 and 18 are EDT pulser, and pulse.AND.trigger, but this is no longer true.
7th  Bank	0xceb80020	Normalization Data not gated by helicity   channel 1=T1, 2=T2, 3 Upstream(x10)BCM, 4=MLU-Strobe, 5=Hel+ pulses, 6=Hel- pulses, 7=Upstream(x3)BCM, 8=1024Hz-clock, 9=Unser, 10=Downstream(x3)BCM, 11=Downstream(x10)BCM, 12=blank, 13=TS-accept(IF R-arm TS), 14=Upstream(x1)BCM, 15=EDT-Pulser, 16=Downstream(x1)BCM. (Note: for e94104 and e98108, chans 1-16 were in 17-32 and were not entirely active. Main norm data was on L-arm).
8th  Bank	0xceb90020	Minus Helicity gated normalization data. Channels 1-32 follow the same map as channels 1-32 of the ceb8 scaler.

The scalers are cleared at the beginning of a run, and read out finally at the end of the run. The timing of the gates, which enables the scalers to count, is such that the end-of-run scaler readout is done after gates are turned off, and then after this final readout the gates are re-enabled, so that scalers can count in between runs. The end-of-run scaler data are written automatically to a file on adaqs2 on the adev account: /home/adev/scaler/scaler_history.dat. Routines exist for reading this file and pulling out the particular scaler channel.

Scaler data are also inserted into the data stream as an event type 140 typically every 4 seconds, but not synchronized to any other event. One way to synchronize them is to use the clock which exists in both the scaler data and the physics events. The time interval between scaler updates has varied in history. After July 1999 the capability to read scalers at 30 Hz with zero deadtime has existed. This allows us to run with the helicity delayed if necessary. However, at present, the data are still integrated and inserted at 4 second intervals.

•   BPM and Raster Data (Left Arm ROC14)

0xfadcb0b4  0x00000002  0xfadc3123  0x0000fc6a  0x000000d4  0x000000c9  
0x0000fb66  0x0000ff35  0x0000ff47  0x0000ff3a  0x0000ff52  0x0000ce1a 
0x0000cd0a  0x0000cd8d  0x0000ce07  0x0000cd3a  0x0000cc37  0x0000cbdd 
0x0000cf43  0xfadc1182  0x000006a2  0x0000096d  0x0000097d  0x00000190  
0x00000cd4  0x0000018f  0x00000757  0x0000018b  0x00000001  0xfadd1182  
0x0000092b  0x00000946  0x0000091c  0x0000090d  0x0000094c  0x00000952  
0x00000980  0x000008d8  0x00000001  0xfca56000  0x00000005  0x00000000  
0x00000000  0x00000000  0x00000000  0x00000000  0x00000000  0x00000000  
0x00000000  0x00000000  0x00000000  0x00000000  0x00000000  0x00000000  
0x00000000  0x00000000  

NOTE: After Nov 10, 1999 there will be additional data. The dump will look different.   For example, here is a dump in May 2001, notice the absence of the VMIC ADC and the presence of two SRUCK 7510 ADCs.

0xfadcb0b4  0xfadc1182  0x00000611  0x00000984  0x00000980  0x0000019d  
0x0000018d  0x00000bf4  0x00000604  0x00000195  0x00000001  0xfadd1182  
0x0000039c  0x0000037b  0x000003e9  0x00000363  0x000003b1  0x00000388 
0x0000037d  0x0000032a  0x00000001  0xfca56000  0x00000002  0x00080fb3 
0x00000000  0xf7510030  0x09640962  0x095c0965  0x0966095e  0x092908ff 
0x09030905  0x09310958  0x09a709d3  0x09e809a9  0x0965096b  0x09140900 
0x0953098e  0x0992097b  0x093a0939  0x09340926  0x09120924  0x09160906  
0x091e092e  0x092b091b  0x08f9090e  0x09180905  0x08ec0910  0x08e208d3  
0x08b20895  0x08d10914  0xf7511030  0x08020803  0x080b0812  0x081d0826  
0x07f307f4  0x07ff0809  0x08180826  0x07db07db  0x07db07db  0x07da07db  
0x07dc07dc  0x07dc07dd  0x07dc07dd  0x04e904ea  0x04ea04ea  0x04e904ea  
0x04f004ef  0x04f004ef  0x04ef04ef  0x04f404f3  0x04f304f3  0x04f304f3  
0x04e604e5  0x04e504e5  0x04e504e6  

In general for VME data, a device is demarked by a unique header. Six possible devices may, or may not, exist: one ADC model VMIC 3123, two ADC model LeCroy1182, one scaler CAEN-V560, and two STRUCK ADC model 7510 ("STR7510"). The STR7510 is new as of Nov 10, 1999, and allows for the ``burst mode'' readout as described below. The headers of these six devices are respectively 0xfadc3123, 0xfadc1182, 0xfadd1182, 0xfca56000, 0xf7510xxx, and 0xf751xxx (where you should assume xxx can be anything, explained later). Following the headers are a number of channels, 16 for VMIC3123, 8 for LeCroy1182, at least 1 and sometimes 16 for the CAEN-V560, and a variable number N for the STR7510 where N is decoded from its lower bits as explained later. One should decode in such a way that any of these devices may be absent, e.g. I may drop ones that are not needed. One should search for the header, then crawl through the bank beyond that header.

The scaler is the easiest: The first channel is a 100 kHz clock, and at present no other channels are used. This clock is a time stamp for the events. (Note: After April 6, 2000 the scaler has an extra word after the header. This word denotes the number of words that follow. E.g. if the word is 2, it means two channels of scaler follow.)

After Dec 17, 2001 I added a time stamp from the variable vxTicks in vxWorks. It comes right after the header for this crate, which is the word 0xfadcb0b4. So it is the 2nd word.

Prior to Nov 10, 1999 the BPM and raster data have been fanned out to both a VMIC3123 and LeCroy1182. After Nov 10, 1999 the STR7510 ADC replaces the VMIC3123, and the signals are still read redundantly by the LeCroy1182. The BPM data are raw signals on antennas that are at 45 degrees to floor. The raster signal is proportinonal to the instantaneous current in the raster. The raster derivative is the sign of the slope of the raster current. It is high or low.

An event pulse is put into one channel of the LeCroy1182 ADC. It is a signal that increments in sawtooth fashion for each event. This signal goes to one ADC in each crate (also fastbus), to verify synchronization.

A note about some old history on the LeCroy ADC with header fadd1182. It existed in the datastream from about Sept 1, 1998 until now with the exception of the time period Nov 20, 1998 - Jan 1, 1999. At 16:00 Nov 20, 1998 we had to take it out of the ROC14 to move it to ROC15 because we had only two such ADCs. LeCroy finally delivered, so that now both ROC14 and ROC15 have two 1182 ADCs, enough to make all twelve signals read redunadantly by the two kinds of ADCs. This note only affects experiment e94010.

Why we read the same data with two kinds of ADCs: The VMIC3123 had the advantage that it takes a signal +/- 5V straight from the electronics, and this ADC had been recommended by some engineers in ACC division. But it shows a problem that occassionally (quite rarely) it gets out of synch !! The LeCroy ADC apparently never gets out of synch, but we've sometimes had trouble adjusting the signal so that under no circumstances does it saturate. After Jan 1, 1999 the LeCroys don't saturate (I think). After Nov 10, 1999 we use the STR7510 instead of VMIC3123. The mapping of the channels for ROC14 is given below.

-------------  Beam Position Monitors 3A and 3B -------------------

  BPM Data       VMIC3123 channel   LeCroy 1182 Channel   STR7510 Channel
antenna raw data                                          (new, Nov 10, 1999)
(X1,Y1 at BPM3A
X2,Y2 at BPM3B)
                                    (header)              (header)
   X1+                     9        (0xfadd1182)   1      (0xf7510xxx)  1
   X1-                    10          "    "       2         "  "       2
Y1+,Y1-,X2+,X2-,Y2+,Y2-   11-16       "    "     3-8         "  "     3-8

 Event Pulse       Not plugged in   (0xfadc1182)   6      Not plugged in


------------------  Raster Data  ----------------------------------

 Raster Data     VMIC3123 channel   LeCroy 1182 Chan   STR7510 Chan

 X-current              1        (0xfadc1182)   7     (0xf7511xxx)  1
 Y-current              4          "    "       1       "    "      2
 X-derivative           2          "    "       2       "    "      3
 Y-derivative           3          "    "       3       "    "      4

 From Nov 20, 1998 until Jan 1, 1999 the fadd1182 ADC was relocated
 in ROC15 crate.  So, during this time we move BPM#2 into fadc1182
 in the following channels (and BPM#1 was not read out):

   BPM#2 Data        VMIC3123 channel       LeCroy 1182 Channel
                        (unchanged)

       X2+                       13                         4
       X2-                       14                         5
       Y2+                       15                         6
       Y2-                       16                         8

The channel ordering is admittedly wierd.  But once it was done 
(by a nameless person), it was less confusing to leave it.

Here is information about the ``burst mode'' readout of the 
BPM and Raster which exists after Nov 10, 1999.  The STR7510 ADC 
allows to read on each event N pulses separated in time by 
T microseconds.  Typically we choose to read N=6 pulses at 
T=4 microsec intervals which gives a snapshot of both the 
amplitude and phase of the signals.  For tests N and T may 
be arranged quite differently (see Bob Michaels for details).  
Below is a bit of C++ code showing the decoding.

    header14str[0] = 0xf7510000;  
    header14str[1] = 0xf7511000;  

// ...imbedded in a loop over event buffer:
    for (k=0; k< NUM7510; k++) {
       if((rdata.evbuffer[ipt]&0xfffff000)==header14str[k]) {
         itemp = (rdata.evbuffer[ipt]&0xfff)/8;
// nhit=number of hits, will always be EVEN, typically 6.
         nhit = (itemp < HITDEPTH) ? itemp : HITDEPTH;  //HITDEPTH=array size.
// loop over 8 channels
	 for(m=0; m<8; m++) {
            khit = 0;
            for(j=0; j< nhit/2; j++) {  
                ipt++;
                itemp = rdata.evbuffer[ipt];
// k=index of ADC unit; m=index of channel inside unit; khit=hit index.
                str7510_raw[k][m][khit++]=(int)((itemp&0x0fff0000)>>16);
                str7510_raw[k][m][khit++]=(int)(itemp&0xfff);
            }
         }
      }
    }

•   BPM and Raster Data (ROC15)

Here is a typical raw data dump of ROC15 (post April 23, 2001)

0xfadcb0b5  0x00000018  0xfadd3123  0x0000f809  0x0000f5f0  
0x0000f42e  0x0000f3dc  0x0000f8d5  0x0000f85f  0x0000f622  
0x0000f62d  0x0000ff32  0x0000ff5f  0x000000f4  0x000000ed  
0x0000ff3a  0x0000ff41  0x0000ff76  0x0000ff45  0xfade1182  
0x00000639  0x0000063a  0x000006cd  0x00000608  0x00000286  
0x00000281  0x000002fd  0x00000302  0xfadf1182  0x00000309  
0x00000354  0x000002f5  0x00000354  0x00000114  0x0000010d  
0x0000011e  0x00000118  0xfca56000  0x00000001  0x00021e8b  

The detector map for ROC15 is as follows:

Header:  0xfadd3123  (model 3123 ADC).


   Channel       Data

    1-8     X1+,X1-,Y1+,Y1-,X2+,X2-,Y2+,Y2-  
             (see notation for ROC14)
    9       Raster X-current
    10      Raster Y-current
    11      X-derivative
    12      Y-derivative

After Nov 20, 1998 switch to 2-TS setup, there will be a LeCroy 1182 ADC in this crate, for redundancy. Ideally we'd have at least 12 channels, but only 8 were available as of Nov 20, 1998 This is important becaues the VMIC ADC sometimes exhibits synch problems.

Header:  0xfade1182

   Channel       Data

    1         X-raster
    2         Y-raster
    3         X-derivative
    4         Y-derivative
   5,6,7,8    respectively  X2+,X2-,Y2+,Y2-

After Jan 1, 1999 we were able to add a second LeCroy 1182 ADC. We then add the following channels (while the above remains the same)

Header:  0xfadf1182

   Channel       Data
 
   1,2,3,4    respectively  X1+,X1-,Y1+,Y1-
 

New in April 2001 is a timestamp, which is a copy of the 100 kHz pulse in roc14 counted by a scaler.

 header:  0xfca56000 
 Next word is the number of scaler channels that are read out.
 Word after that is the scaler.
 In the above dump, it's 1 channel with data = 0x00021e8b

•  EPICS   INFORMATION

Data from various EPICS databases are periodically inserted into the datastream, as well as placed in text files that are saved at the start-of-run and end-of-run.  These text files are stored as, for example, /home/adev/epics/runfiles/Start_of_Run_1047.epics, and End_of_Run_1047.epics where 1047 is the run number. These data are also written to the electronics log book HALOG.   Each experiment is permitted to modify the script that controls this (ask me) but please let only ONE experienced, qualifed person modify the script (and it is probably good enough, too). Recently Mark Jones has been maintaining this script.  Incidentally, no other script dealing with DAQ should ever be modified.  

The data which are inserted periodically include the beam current, position monitors,  magnetic fields in our spectrometers, as well as some beamline elements, the beam energy, collimator positions, and several other things. Approximately every 30 seconds, a long list of information from the accelerator and from Hall A are inserted as event type 131.  Approximately every 5 seconds a shorter list of information, for which rapid updates are desirable, are inserted as event type 131.  The shorter list contain the BPMs near the target and the charge monitor data.  The synchronization of the data relative to our datastream is good to perhaps 1 second.  

Here is an example of the rapidly updated EPICS data which are inserted every approximately 5 seconds :

Tue Aug 25 12:59:43 EDT 1998
IPM1H03A.XPOS                  0.352823
IPM1H03A.YPOS                  0.430828
IPM1H03B.XPOS                  -0.130145
IPM1H03B.YPOS                  -0.48034
hac_bcm_average                5.29884
hac_bcm_dvm1_current           5.30776
hac_bcm_dvm2_current           5.28991
hac_unser_current              5.30134

Here is a link showing more fully some examples of EPICS data, though some details have changed since that information was posted.  One can see that the data are in the form of  a series of lines of character data, where each line is of the form "keyword  data-value", where keyword is a descriptive word (actually the name of the EPICS variable) and data-value is its numerical value.

•  SPECIAL   EVENTS

Event types less than 15 are physics triggers of various types.  There are also prestart, go, and end "events".  Event types greater than 127 are "special events", which are not really "events" and not particularly "special" either -- they are data inserted into the CODA datastream which are useful to record with the event data.  The following special events exist:

Specially Inserted Events

Event  Type	Format	Description
131	character	EPICS  Data  (see section above)
133	character	Prescale Factors  (decoded by ESPACE)
135	character	Detector Map (optionally used by ESPACE)
136	character	Trigger  Setup  (read it and see)
140	integer	Scaler Data (each spectrometer every few seconds)