HALL A RAW DATA STRUCTURE
R. Michaels, Jefferson Lab, e-mail: email@example.com, updated Jan 6, 2005
The raw data structure recorded in the spectrometer DAQ datastream will be described. This includes the following data: 1) Event data recorded with each accepted trigger (fastbus and VME data from the two spectrometers and from the BPM and raster); 2) scalers; 3) EPICS information; and 4) "special events" which are inserted once or infrequently during a run, such as prescale information and detector map.
Fortunately, there is already a large body of software which decodes and interprets these data, so in many cases the experimenter may use higher level variables, such as presented in ESPACE kumacs. The new ROOT/C++ analyzer also has a decoder.
I will not describe the various other datastreams in Hall A, such as Moller polarimeter, e-P energy measurement, Compton Polarimeter, etc.
For some notes about the year 2001 and a bit beyond see dstruct_year2001.html. To find out about the data prior to the year 2001, see the URL: dstruct_pre2001.html.
CODA DATA FORMAT
The best detailed reference on the CODA format was the old manual for CODA version 1.4, see appendix E. I have saved this postscript file coda1.4.ps. I will give a simplified summary here. The data come in the form of an array of 32-bit words, in the case of event data, or character data, in the case of some of the special events. In general, the first part of the event structure is "header" information, indicating how long the event is, what the event type is, etc. The most important header and other identifying data words are:
|Location in Buffer||Content||Description|
|1st word||event length||
loop from J=1 to J=evbuffer(1)+1 in Fortran
to access data = evbuffer(J)
|ishft( evbuffer(2), -16 )||event type||
Types 1-12 are triggers that don't overlap in
time. Event 5 are coincidences, or any trigger overlap
within 10 nsec. Other types are prestart, go, end, etc.
Types >128 are "special events".
|evbuffer(4) in event type 17||run number||The run number, which is normally in the filename also.|
|evbuffer(5)||event number||The event number, uniquely assigned by CODA|
After the header, the data are organized in banks corresponding to each ROC as of Sept 2002: A ROC is a "readout controller", and it means a crate of fastbus or VME. We have several ROCs in our spectrometer DAQ event stream: 1) ROC1 = fastbus on R-arm; 2) ROC2 = 2nd fastbus on R-arm scintillators; 3) ROC3 = fastbus on L-arm; 4) ROC4 = fastbus on L-arm (FPP crate). and being phased out is 5) ROC14 = VME crate for BPM and raster data. The beginning of each ROC bank contains the length of the bank and an identifying tag to indicate which ROC number. A bit of Fortran code which illustrates how to crawl through the banks is shown below.
do 150 i = 1,MAXROC c n1 = pointer to first word of ROC if(i.eq.1) then n1 = evbuffer(3) + 4 else n1 = n1roc(irn(i-1))+lenroc(irn(i-1))+1 endif c irn = ROC number obtained from CODA irn(i) = ishft(iand(evbuffer(n1+1),'ff0000'x),-16) if(irn(i).le.0.or.irn(i).gt.maxroc) then write(6,*) 'fatal error in decoding' stop endif c Store pointers and lengths if(i.eq.1) then n1roc(irn(1)) = n1 lenroc(irn(1)) = evbuffer(n1) lentot = n1 + evbuffer(n1) else n1roc(irn(i)) = n1roc(irn(i-1))+lenroc(irn(i-1))+1 lenroc(irn(i)) = evbuffer(n1roc(irn(i))) lentot = lentot + lenroc(irn(i)) + 1 endif if(lentot.ge.len-1) goto 200 150 end do 200 numroc = i
Libraries exist for reading and writing CODA data format. The library is in the ESPACE distribution, in the /evio directory. A convenient example code for reading/writing CODA data (written in C) is here: rwcoda.c with a Makefile (one can link to CODA libraries or simply compile evio.c and link it instead).
If you want to be modern and hip, try my CODA classes (C++ classes for CODA interface). This and the hana decoder have been integrated into the new ROOT/C++ analyzer
The data are decoded by ESPACE in the routine event_decoder which is in the module coda.f. This routine requires the detector map. Normally the ``official detmap'' is kept in ~adaq/detmap directory on the ADAQ cluster. This detector map gives the correspondence between the "raw channel number" and the "analysis detector channel". By "raw channel number" I mean the combination of : (ROC number, slot in ROC, channel within the slot). By "analysis detector channel" I mean the channel within a detector as indexed by ESPACE. The detector map is also stored in the CODA file as a special event and can be used by ESPACE by leaving out (or commenting out) the line "set/file/detmap detmap.config" in the ESPACE kumac. The absence of this line causes ESPACE to search the CODA file for the detmap. The presence of this line causes ESPACE to take the detmap specified in the kumac, which then over-rides whatever is written in the CODA file. The intention, of course, is that the detmap in the CODA file is the correct one at the time of the experiment. There have been some reports that ESPACE occassionally (rarely) fails to find the detmap in a CODA file. You can work around this problem by extracting it once and then renaming this file and using it as the the file pointed to in the kumac file. The extracted detmap file is always called detector_map.lis.
Each crate adds some ``event flag'' data, which are extra words that don't come from a module. The set of flags that exist has changed over time, and I will give a snapshot that is valid as of Thurs Jan 6, 13:00 EST 2005. These data consist of one or several words after a "header". A header is a unique flag which is impossible to misinterpret as data from a module.
In Fastbus: 0xfabc0006 -- Header (hex format) denoting the start of event flags. Because of the "f" this is impossible to confuse with fastbus data. The lowest 3 bits tell how many flags follow, in this case 6. event count -- Event counter (locally counted in each CRL). This increments to 20000 and is reset to 0. datascan -- Tells the bit pattern of what fastbus crates had a hit event time -- The vxWorks event time. This has 1/60 sec resolution. event type -- This is the event type that the trigger supervisor told to the ROC via a cable connection (hardware). The fastbus 104 kHz clock (was disabled from Dec 2003 - Jan 1, 2005) clock -- A scaler reading 104 kHz clock is available in each crate. To get it from the ROC10/11 data, see section on ROC10/11. Then there is a flag showing the buffering mode 0xfaaa0001 -- Header (0x means hex format) 0xfabb0000 -- If you see this, data are unbuffered. 0xfafbbf00+synchFlag -- If you see this the data are buffered. If synchFlag !=0 then it was a synch event.In ROC14/15 it is a little different.
In ROC14: 0xfadcb0b4 -- Header (0x means hex format) event time -- The vxWorks event time. This has 1/60 sec resolution. event type -- This is the event type that the trigger supervisor told to the ROC via a cable connection (hardware). event count -- Event counter (counted locally in each ROC). Resets to 0 every 20 K events.In Dec 2003 we added the RICH crates ROC12 and 16. Here are some synch flag data:
From the header 0x73540004 there are words event type -- Event type from TS event time -- vxWorks time Crams-TimeOut -- not sure what this is, ask Bodo. event couner (mod 65535). -- Event counter.
Scalers count raw hits on phototubes, as well as important quantities like charge and triggers which are used to normalize the experiment. Scalers appear in the raw data as event type 140 (so-called "scaler events") as well as in the ROC10/11 data (see section below). In addition, the scalers are displayed online using the "xscaler" code, and the meaning of the channels is evident from the scaler configuration file "scaler.config" which that code uses. The "scaler.config" file is normally located in the directory from which you must run "xscaler", namely ~adaq/$EXPERIMENT/right/scaler ~adaq/$EXPERIMENT/left/scaler where $EXPERIMENT is an environment variable like e94104. Note, to run the analyzer you also need a "scaler.map" file as explained at hallaweb.jlab.org/equipment/daq/THaScaler.html
The scalers have been pretty stable in physical layout for years. In the long shutdown of Fall 2003 I decided to add a bunch of scalers and to rearrange the helicity gated scalers so that the only redundancy was between two spectrometers; this liberated some channels. The information here is up-to-date for Oct 2003 and later. For arrangements of scalers prior to Oct 2003 please read scaler_preOct2003.html.
The headers are of the form "0xcebN00XY" for R-arm scalers and "0xabcN00XY" for L-arm scalers where N is the "software slot" and XY encodes the number of scalers channels in the lowest 6 bits (16 or 32 channels). The software slot is an arbitrary index used by various software and for historical reasons it may not be the hardware slot. It may seem confusing that for example 0xceb7 (software slot 7) is the presently the 2nd hardware slot, but there are reasons -- I wanted to move these scalers out of the way, to reduce the chance of bumping cables, but keeping this software slot number was preferable to breaking someone's old code. The helicity gated scalers are FIFO scalers run in G0 helicity mode, see also www.jlab.org/~rom/g0helicity.html. The helicity scalers appear in the data stream in event type 140 already sorted by helicity (the sorting is done online) but also in raw form; see the section below on ROC10/11.
|Location in Buffer||Slot in Crate (Left to Right)||16 or 32 Chan||Header (see notes above)||Scaler Model||USAGE|
|1||1||32||0xceb00020 (software slot 0)||SIS3800||1-6=S1 Left PMTs, 7,8=blank, 9-14=S1 Right PMTs, 17-22 = Left.and.Right S1|
|2||2||32||0xceb70020 (traditionally slot 7)||SIS3801||Plus Helicity gated normalization data. Data from one scaler device are sorted by helicity on the fly. Channels 1-32 follow the same map as channels 1-32 of the ceb8 scaler.|
|3||3||32||0xceb80020 (traditionally slot 8)||SIS3800||Normalization Data not gated by helicity channel 1=T1, 2=T2, 3=Upstream(x10)BCM, 4=MLU-Strobe, 5=Hel+ pulses, 6=Hel- pulses, 7=Upstream(x3)BCM, 8=1024Hz-clock, 9=Unser, 10=Downstream(x3)BCM, 11=Downstream(x10)BCM, 12=blank, 13=TS-accept(IF R-arm TS), 14=Upstream(x1)BCM, 15=EDT-Pulser, 16=Downstream(x1)BCM. 18 = helicity transitions, 19 = QRT, 20 = MPS, 21 = LNE, 22=T3 from L-arm, 23=T4 from L-arm.|
|4||2 (same scaler unit as ceb7)||32||0xceb90020 (traditionally slot 9)||SIS3801||Minus Helicity gated normalization data. Data from one scaler device are sorted by helicity on the fly. Channels 1-32 follow the same map as channels 1-32 of the ceb8 scaler.|
|5||4||32||0xceb10020 (software slot 1)||SIS3800||S2m PMTs|
|6||5||16||0xceb20010||LeCroy 1151||Cerenkov's (1-10 PMTs, 11=Sum).|
|Location in Buffer||Slot in Crate (Left to Right)||16 or 32 Chan||Header (see notes above)||Scaler Model||USAGE|
|1||1||16||0xabc00010 (software slot 0)||LeCroy 1151||1-6=S1 Left PMTs, 7,8=blank, 9-14=S1 Right PMTs|
|2||2||16||0xabc10010 (software slot 1)||LeCroy 1151||1-6=S1 Left.and.Right PMTs|
|4||4||32||0xabc30020||SIS3801||Plus Helicity gated normalization data. Data from one scaler device are sorted by helicity on the fly. Channels 1-32 follow the same map as channels 1-32 of the abc4 scaler.|
|5||5||32||0xabc40020||SIS3800||Normalization Data not gated by helicity. Channel 1=T1, 2=T2, 3=T3, 4=T4, 5=T5, 6=Hel+ Pulses, 7=Upstream(x3)BCM, 8=1024Hz-clock, 9=Upstream(x10)BCM, 10=Hel- Pulses, 11=Downstream(x3)BCM, 12=Downstream(x10)BCM, 13=Accepted-Triggers(If Trig Super on L-arm), 14=Upstream(x1)BCM, 15=Unser, 16=Downstream(x1)BCM), 17=Q10-gated-by-busy, 18=clock-gated-by-busy, 20=retiming pulse (``or'' of Right PMTs), 24=1024 clock (again), 25 = MLU output 1, 26 = helicity, 27 = QRT, 28 = MPS, 29 = LNE|
|6||4 (same scaler unit as abc3)||32||0xabc50020||SIS3801||Minus Helicity gated normalization data. Data from one scaler device are sorted by helicity on the fly. Channels 1-32 follow the same map as channels 1-32 of the abc4 scaler.|
|10||9||16||0xabc90010||CAEN 560||Misc signals. 16 = 105 kHz clock.|
The scalers are cleared at the beginning of a run, and read out finally at the end of the run. The timing of the gates, which enables the scalers to count, is such that the end-of-run scaler readout is done after gates are turned off, and then after this final readout the gates are re-enabled, so that scalers can count in between runs. The end-of-run scaler data are written automatically to a file on ~a-onl/scaler/scaler_history.dat. Routines exist for reading this file and pulling out the particular scaler channel.
Scaler data are also inserted into the data stream as an event type 140 typically every 4 seconds, but not synchronized to any other event. The capability to read scalers at 30 Hz with zero deadtime is also supported, and these data appear as ROC10 and ROC11, see section below. New in May 2003, an event type 100 is written at end of run which has same format as ROC10/11 which is the final scaler reading from ROC10/11. Also new are the "TS2 scalers" which are scalers on the new trigger supervisor, see ROC10/11 section below.
TS2 SCALERS : The new trigger supervisor TS2 has scalers on board. This "TS2 scaler" data was available after March 31, 2003. The data starts with header 0xfed00015 in the ROC10 ring buffer readout (see scaler_roc10.html link above). Firstly, the 0x15 in lower bits encodes how many data follows. These data are the 21 scaler data from TS2 registers. Here is the format:
scaler 1 = trigger 1 input, i.e. after prescaling 2 = trigger 2 3 = trigger 3 same 4,5,6... 12 13 = "OR" of triggers 14 = Level 1 accept (can use this as alternative to external scaler L1A) 15 = scheduled sync's 16 = programmed events (normally zero unless I program special events) 17 = latched triggers 18 = program 2 events After these 18 scalers are 3 more data that come from TS2 registers: 19th = Event count 20th = Live 1 count (see below) 21st = Live 2 count --------------- Live 1 count = free running 200 kHz clock gated by DAQ live signal (.NOT. busy) Live 2 count = same clock, no gate. Live Time = (Live 1 count) / (Live 2 count) <= 1.0 (However, this live1/live2 ratio only works if psfactor=1. If psfactor!=1 it is wrong because it incorrectly assumes random trigger.) The TS2 scalers are latched at start of ring buffer readout to avoid time slewing. The latch is freed after a read, but the latching doesn't affect the actual counting.
The BPM/raster data are in fastbus. See the detector map in ~adaq/detmap/detmap.config.
One issue is that one must obtain good pedestals for BPMs. This is done by turning the beam off and asking MCC to set the BPMs to forced gain with gain setting zero as follows:
-- Open BPM window "BPM Diagnostics - SEE" -- From there pull down the "Expert Screens" and open "Gain Control" -- For IOCSE10 (this might change) use the pull down window and change from "auto gain" to "forced gain". Then change the Forced Gain values x and y to zero. -- Do 10 min CODA run with ps8=4. -- Remember to put back "autogain" when you are finished. -- Pedestals will appear as extremely narrow peaks in the raw histograms available when you run spot (do ".ls" at root prompt to see the list of raw ADC histograms). They are NOT the same as "ordinary" fastbus pedestals.
Data from various EPICS databases are periodically inserted into the datastream, as well as placed in text files that are saved at the start-of-run and end-of-run. These text files are stored as, for example, ~a-onl/epics/runfiles/Start_of_Run_1047.epics, and End_of_Run_1047.epics where 1047 is the run number. These data are also written to the electronics log book HALOG. Each experiment is permitted to modify the script that controls this (ask me) but please let only ONE experienced, qualifed person modify the script (and it is probably good enough, too). Incidentally, no other dealing with DAQ should ever be modified without asking me. I leave things open but I must be informed.
The data which are inserted periodically include the beam current, position monitors, magnetic fields in our spectrometers, as well as some beamline elements, the beam energy, collimator positions, and several other things. Approximately every 30 seconds, a long list of information from the accelerator and from Hall A are inserted as event type 131. Approximately every 5 seconds a shorter list of information, for which rapid updates are desirable, are inserted as event type 131. The shorter list contain the BPMs near the target and the charge monitor data. The synchronization of the data relative to our datastream is good to perhaps 1 second.
Here is an fictitious example of the rapidly updated EPICS data which are inserted every approximately 5 seconds :
Tue Aug 27 12:59:43 EDT 2002 IPM1H04A.XPOS 0.352823 IPM1H04A.YPOS 0.430828 IPM1H04B.XPOS -0.130145 IPM1H04B.YPOS -0.48034 hac_bcm_average 5.29884 hac_bcm_dvm1_current 5.30776 hac_bcm_dvm2_current 5.28991 hac_unser_current 5.30134
There is some code in the C++ analyzer and ESPACE to pull out the data. However, since the EPICS "event" data are characters they are viewable simply with grep. From a Linux box you can type "grep -a IPM1H0 e01012_1455.dat.0" on the data file, and from SunOS you don't need the "-a". grep is case sensitive unless you use "-i" option (so "grep -ai ipm filename", etc).
Event types less than 15 are physics triggers of various types. There are also prestart, go, and end "events". Event types greater than 99 are "special events", which are not really "events" and not particularly "special" either -- they are data inserted into the CODA datastream which are useful to record with the event data. The following special events exist:
|100||integer||Last scaler reading (same format as ROC10/11).|
|120||integer||Prescale registers (8 numbers read from TS)|
|131||character||EPICS Data (see section above)|
|133||character||Prescale Factors (decoded by ESPACE)|
|135||character||Detector Map (optionally used by ESPACE)|
|136||character||Trigger Setup (read it and see)|
|140||integer||Scaler Data (each spectrometer every few seconds)|