HALL  A  RAW  DATA  STRUCTURE

R. Michaels,  Jefferson Lab,  e-mail:  rom@jlab.org,  updated Jan 6, 2005

•  OVERVIEW

The raw data structure recorded in the spectrometer DAQ datastream will be described.  This includes the following data:  1) Event data recorded with each accepted trigger  (fastbus and VME data from the two spectrometers and from the BPM and raster);  2) scalers;  3)  EPICS information;  and 4)  "special events" which are inserted once or infrequently during a run, such as prescale information and detector map.

Fortunately, there is already a large body of software which decodes and interprets these data, so in many cases the experimenter may use higher level variables, such as presented in ESPACE kumacs. The new ROOT/C++ analyzer also has a decoder.

I will not describe the various other datastreams in Hall A, such as Moller polarimeter,  e-P energy measurement,  Compton Polarimeter, etc.

For some notes about the year 2001 and a bit beyond see dstruct_year2001.html.     To find out about the data prior to the year 2001, see the URL:   dstruct_pre2001.html.

•  CODA  DATA  FORMAT

The best detailed reference on the CODA format was the old manual for CODA version 1.4, see appendix E.   I have saved this postscript file coda1.4.ps.   I will give a simplified summary here.  The data come in the form of an array of 32-bit words, in the case of event data, or character data, in the case of some of the special events.  In general, the  first part of the event structure is "header" information, indicating how long the event is, what the event type is, etc.  The most important header and other identifying data words are:

Identifying CODA Data Words

Location in Buffer	Content	Description
1st  word	event length	loop from J=1 to J=evbuffer(1)+1 in Fortran to access data = evbuffer(J)
ishft( evbuffer(2), -16 )	event  type	Types 1-12 are triggers that don't overlap in time.   Event 5 are coincidences, or any trigger overlap within 10 nsec.  Other types are prestart, go, end, etc. Types >128 are "special events".
evbuffer(4) in event type 17	run number	The run number, which is normally in the filename also.
evbuffer(5)	event number	The event number, uniquely assigned by CODA

After the header, the data are organized in banks corresponding to each ROC as of Sept 2002:   A ROC is a "readout controller", and it means a crate of fastbus or VME.   We have several ROCs in our spectrometer DAQ event stream:  1) ROC1 = fastbus on R-arm;  2) ROC2 = 2nd fastbus on R-arm scintillators; 3) ROC3 = fastbus on L-arm;   4) ROC4 = fastbus on L-arm (FPP crate). and being phased out is 5) ROC14 = VME crate for BPM and raster data.  The beginning of each ROC bank contains the length of the bank and an identifying tag to indicate which ROC number.  A bit of Fortran code which illustrates how to crawl through the banks is shown below.

          do 150 i = 1,MAXROC
c n1 = pointer to first word of ROC
            if(i.eq.1) then
               n1 = evbuffer(3) + 4
            else
               n1 = n1roc(irn(i-1))+lenroc(irn(i-1))+1
            endif
c irn = ROC number obtained from CODA
            irn(i) = ishft(iand(evbuffer(n1+1),'ff0000'x),-16)
            if(irn(i).le.0.or.irn(i).gt.maxroc) then
               write(6,*) 'fatal error in decoding'
               stop
            endif
c Store pointers and lengths
            if(i.eq.1) then
               n1roc(irn(1)) = n1
               lenroc(irn(1)) = evbuffer(n1)
               lentot = n1 + evbuffer(n1)
            else
               n1roc(irn(i)) = n1roc(irn(i-1))+lenroc(irn(i-1))+1
               lenroc(irn(i)) = evbuffer(n1roc(irn(i)))
               lentot = lentot + lenroc(irn(i)) + 1
            endif
            if(lentot.ge.len-1) goto 200
150     end do
200     numroc = i

Libraries exist for reading and writing CODA data format. The library is in the ESPACE distribution, in the /evio directory. A convenient example code for reading/writing CODA data (written in C) is here: rwcoda.c with a Makefile (one can link to CODA libraries or simply compile evio.c and link it instead).

If you want to be modern and hip, try my   CODA classes   (C++ classes for CODA interface). This and the hana decoder have been integrated into the new ROOT/C++ analyzer

•   DECODING  OF  DATA

The data are decoded by ESPACE in the routine event_decoder which is in the module coda.f.   This routine requires the detector map.   Normally the ``official detmap'' is kept in ~adaq/detmap directory on the ADAQ cluster.   This detector map gives the correspondence between the "raw channel number" and the "analysis detector channel".  By "raw channel number" I mean the combination of :  (ROC number, slot in ROC,  channel within the slot).  By "analysis detector channel"  I mean the channel within a detector as indexed by ESPACE.  The detector map is also stored in the CODA file as a special event and can be used by ESPACE by leaving out (or commenting out)  the line "set/file/detmap detmap.config" in the ESPACE kumac.  The absence of this line causes ESPACE to search the CODA file for the detmap.  The presence of this line causes ESPACE to take the detmap specified in the kumac, which then over-rides whatever is written in the CODA file.  The intention, of course, is that the detmap in the CODA file is the correct one at the time of the experiment.  There have been some reports that ESPACE occassionally (rarely) fails to find the detmap in a CODA file.  You can work around this problem by extracting it once and then renaming this file and using it as the the file pointed to in the kumac file.  The extracted detmap file is always called detector_map.lis.

•  EVENT   FLAGS

Each crate adds some ``event flag'' data, which are extra words that don't come from a module. The set of flags that exist has changed over time, and I will give a snapshot that is valid as of Thurs Jan 6, 13:00 EST 2005. These data consist of one or several words after a "header". A header is a unique flag which is impossible to misinterpret as data from a module.

In Fastbus:
   0xfabc0006  -- Header (hex format) denoting the start of event flags.
                  Because of the "f" this is impossible to confuse
                  with fastbus data.  The lowest 3 bits tell how
                  many flags follow, in this case 6.
   event count -- Event counter (locally counted in each CRL).
                  This increments to 20000 and is reset to 0.
   datascan    -- Tells the bit pattern of what fastbus crates
                  had a hit
   event time  -- The vxWorks event time.  This has 1/60 sec resolution.
   event type  -- This is the event type that the trigger supervisor
                  told to the ROC via a cable connection (hardware).
The fastbus 104 kHz clock (was disabled from Dec 2003 - Jan 1, 2005)
   clock       -- A scaler reading 104 kHz clock is available in 
                  each crate.  To get it from the ROC10/11 data, see
                  section on ROC10/11. 
Then there is a flag showing the buffering mode
   0xfaaa0001  -- Header (0x means hex format)
   0xfabb0000  -- If you see this, data are unbuffered.
   0xfafbbf00+synchFlag -- If you see this the data are buffered.
                  If synchFlag !=0 then it was a synch event.

In ROC14:
   0xfadcb0b4 -- Header (0x means hex format)
   event time  -- The vxWorks event time.  This has 1/60 sec resolution.
   event type  -- This is the event type that the trigger supervisor
                  told to the ROC via a cable connection (hardware).
   event count -- Event counter (counted locally in each ROC).
                  Resets to 0 every 20 K events.

From the header 0x73540004 there are words
   event type  -- Event type from TS
   event time  -- vxWorks time
   Crams-TimeOut -- not sure what this is, ask Bodo.
   event couner (mod 65535).  -- Event counter.

•   SCALERS

Scalers count raw hits on phototubes, as well as  important quantities like charge and triggers which are used to normalize the experiment.   Scalers appear in the raw data as event type 140 (so-called "scaler events") as well as in the ROC10/11 data (see section below).   In addition, the scalers are displayed online using the "xscaler" code, and the meaning of the channels is evident from the scaler configuration file "scaler.config" which that code uses.  The "scaler.config" file is normally located in the directory from which you must run "xscaler", namely ~adaq/$EXPERIMENT/right/scaler ~adaq/$EXPERIMENT/left/scaler where $EXPERIMENT is an environment variable like e94104.   Note, to run the analyzer you also need a "scaler.map" file as explained at hallaweb.jlab.org/equipment/daq/THaScaler.html

The scalers have been pretty stable in physical layout for years. In the long shutdown of Fall 2003 I decided to add a bunch of scalers and to rearrange the helicity gated scalers so that the only redundancy was between two spectrometers; this liberated some channels. The information here is up-to-date for Oct 2003 and later. For arrangements of scalers prior to Oct 2003 please read scaler_preOct2003.html.

The headers are of the form "0xcebN00XY" for R-arm scalers and "0xabcN00XY" for L-arm scalers where N is the "software slot" and XY encodes the number of scalers channels in the lowest 6 bits (16 or 32 channels). The software slot is an arbitrary index used by various software and for historical reasons it may not be the hardware slot. It may seem confusing that for example 0xceb7 (software slot 7) is the presently the 2nd hardware slot, but there are reasons -- I wanted to move these scalers out of the way, to reduce the chance of bumping cables, but keeping this software slot number was preferable to breaking someone's old code. The helicity gated scalers are FIFO scalers run in G0 helicity mode, see also www.jlab.org/~rom/g0helicity.html. The helicity scalers appear in the data stream in event type 140 already sorted by helicity (the sorting is done online) but also in raw form; see the section below on ROC10/11.

Arrangement of  the Right   Spectrometer   Valid after Oct 2003

Location in Buffer	Slot in Crate (Left to Right)	16 or 32 Chan	Header (see notes above)	Scaler Model	USAGE
1	1	32	0xceb00020   (software slot 0)	SIS3800	1-6=S1 Left PMTs, 7,8=blank, 9-14=S1 Right PMTs, 17-22 = Left.and.Right S1
2	2	32	0xceb70020   (traditionally slot 7)	SIS3801	Plus Helicity gated normalization data. Data from one scaler device are sorted by helicity on the fly. Channels 1-32 follow the same map as channels 1-32 of the ceb8 scaler.
3	3	32	0xceb80020   (traditionally slot 8)	SIS3800	Normalization Data not gated by helicity   channel 1=T1, 2=T2, 3=Upstream(x10)BCM, 4=MLU-Strobe, 5=Hel+ pulses, 6=Hel- pulses, 7=Upstream(x3)BCM, 8=1024Hz-clock, 9=Unser, 10=Downstream(x3)BCM, 11=Downstream(x10)BCM, 12=blank, 13=TS-accept(IF R-arm TS), 14=Upstream(x1)BCM, 15=EDT-Pulser, 16=Downstream(x1)BCM. 18 = helicity transitions, 19 = QRT, 20 = MPS, 21 = LNE, 22=T3 from L-arm, 23=T4 from L-arm.
4	2 (same scaler unit as ceb7)	32	0xceb90020   (traditionally slot 9)	SIS3801	Minus Helicity gated normalization data. Data from one scaler device are sorted by helicity on the fly. Channels 1-32 follow the same map as channels 1-32 of the ceb8 scaler.
5	4	32	0xceb10020   (software slot 1)	SIS3800	S2m PMTs
6	5	16	0xceb20010	LeCroy 1151	Cerenkov's (1-10 PMTs, 11=Sum).
7	6	16	0xceb30010	LeCroy 1151	EDTM
8	7	16	0xceb40010	CAEN 560	spare
9	8	16	0xceb50010	CAEN 560	spare
10	9	16	0xceb60010	CAEN 560	spare

The scalers are cleared at the beginning of a run, and read out finally at the end of the run. The timing of the gates, which enables the scalers to count, is such that the end-of-run scaler readout is done after gates are turned off, and then after this final readout the gates are re-enabled, so that scalers can count in between runs. The end-of-run scaler data are written automatically to a file on ~a-onl/scaler/scaler_history.dat. Routines exist for reading this file and pulling out the particular scaler channel.

Scaler data are also inserted into the data stream as an event type 140 typically every 4 seconds, but not synchronized to any other event. The capability to read scalers at 30 Hz with zero deadtime is also supported, and these data appear as ROC10 and ROC11, see section below. New in May 2003, an event type 100 is written at end of run which has same format as ROC10/11 which is the final scaler reading from ROC10/11. Also new are the "TS2 scalers" which are scalers on the new trigger supervisor, see ROC10/11 section below.

•   Trigger Supervisor Crates (ROC10 or ROC11, and TS2 Scalers)

TS2 SCALERS :     The new trigger supervisor TS2 has scalers on board. This "TS2 scaler" data was available after March 31, 2003. The data starts with header 0xfed00015 in the ROC10 ring buffer readout (see scaler_roc10.html link above). Firstly, the 0x15 in lower bits encodes how many data follows. These data are the 21 scaler data from TS2 registers. Here is the format:

scaler 1 = trigger 1 input, i.e. after prescaling
2 = trigger 2
3 = trigger 3
same 4,5,6... 12
13 = "OR" of triggers
14 = Level 1 accept (can use this as alternative to external scaler L1A)
15 = scheduled sync's
16 = programmed events (normally zero unless I program special events)
17 = latched triggers
18 = program 2 events
After these 18 scalers are 3 more data that come from TS2 registers:
19th = Event count
20th = Live 1 count (see below)
21st = Live 2 count
---------------
Live 1 count = free running 200 kHz clock gated by DAQ live signal (.NOT. busy)
Live 2 count = same clock, no gate.
Live Time = (Live 1 count) / (Live 2 count) <= 1.0
(However, this live1/live2 ratio only works if psfactor=1.  
If psfactor!=1 it is wrong because it incorrectly assumes random trigger.)

The TS2 scalers are latched at start of ring buffer readout 
to avoid time slewing.  The latch is freed after a read, but 
the latching doesn't affect the actual counting.

•   BPM and Raster Data (ROC14 or ROC15)

The BPM/raster data are in fastbus. See the detector map in ~adaq/detmap/detmap.config.

One issue is that one must obtain good pedestals for BPMs. This is done by turning the beam off and asking MCC to set the BPMs to forced gain with gain setting zero as follows:

-- Open BPM window "BPM Diagnostics - SEE"
-- From there pull down the "Expert Screens" and open "Gain Control"
-- For IOCSE10 (this might change) use the pull down window
   and change from "auto gain" to "forced gain".   Then change the 
   Forced Gain values x and y to zero.
-- Do 10 min CODA run with ps8=4.
-- Remember to put back "autogain" when you are finished.  
-- Pedestals will appear as extremely narrow peaks in the raw histograms
available when you run spot (do ".ls" at root prompt to see the list of
raw ADC histograms).  They are NOT the same as "ordinary" fastbus pedestals.

•  EPICS   INFORMATION

Data from various EPICS databases are periodically inserted into the datastream, as well as placed in text files that are saved at the start-of-run and end-of-run.  These text files are stored as, for example, ~a-onl/epics/runfiles/Start_of_Run_1047.epics, and End_of_Run_1047.epics where 1047 is the run number. These data are also written to the electronics log book HALOG.   Each experiment is permitted to modify the script that controls this (ask me) but please let only ONE experienced, qualifed person modify the script (and it is probably good enough, too).  Incidentally, no other dealing with DAQ should ever be modified without asking me. I leave things open but I must be informed.

The data which are inserted periodically include the beam current, position monitors,  magnetic fields in our spectrometers, as well as some beamline elements, the beam energy, collimator positions, and several other things. Approximately every 30 seconds, a long list of information from the accelerator and from Hall A are inserted as event type 131.  Approximately every 5 seconds a shorter list of information, for which rapid updates are desirable, are inserted as event type 131.  The shorter list contain the BPMs near the target and the charge monitor data.  The synchronization of the data relative to our datastream is good to perhaps 1 second.  

Here is an fictitious example of the rapidly updated EPICS data which are inserted every approximately 5 seconds :

Tue Aug 27 12:59:43 EDT 2002
IPM1H04A.XPOS                  0.352823
IPM1H04A.YPOS                  0.430828
IPM1H04B.XPOS                  -0.130145
IPM1H04B.YPOS                  -0.48034
hac_bcm_average                5.29884
hac_bcm_dvm1_current           5.30776
hac_bcm_dvm2_current           5.28991
hac_unser_current              5.30134

There is some code in the C++ analyzer and ESPACE to pull out the data. However, since the EPICS "event" data are characters they are viewable simply with grep. From a Linux box you can type "grep -a IPM1H0 e01012_1455.dat.0" on the data file, and from SunOS you don't need the "-a".   grep is case sensitive unless you use "-i" option (so "grep -ai ipm filename", etc).

•  SPECIAL   EVENTS

Event types less than 15 are physics triggers of various types.  There are also prestart, go, and end "events".  Event types greater than 99 are "special events", which are not really "events" and not particularly "special" either -- they are data inserted into the CODA datastream which are useful to record with the event data.  The following special events exist:

Specially Inserted Events

Event  Type	Format	Description
100	integer	Last scaler reading (same format as ROC10/11).
120	integer	Prescale registers (8 numbers read from TS)
131	character	EPICS  Data  (see section above)
133	character	Prescale Factors  (decoded by ESPACE)
135	character	Detector Map (optionally used by ESPACE)
136	character	Trigger  Setup  (read it and see)
140	integer	Scaler Data (each spectrometer every few seconds)