HALL A RAW DATA STRUCTURE PRIOR TO 2001
R. Michaels, Jefferson Lab, e-mail: rom@jlab.org, last updated Nov 1, 2000
The raw data structure recorded in the spectrometer DAQ datastream will be described. This includes the following data: 1) Event data recorded with each accepted trigger (fastbus and VME data from the two spectrometers and from the BPM and raster); 2) scalers; 3) EPICS information; and 4) "special events" which are inserted once or infrequently during a run, such as prescale information and detector map.
Fortunately, there is already a large body of software which decodes and interprets these data, so in many cases the experimenter may use higher level variables, such as presented in ESPACE kumacs. In addition to ESPACE, there are some auxiliary routines (from L. Todor, for example) which interpret the scalers and EPICS data.
I will not describe the various other datastreams in Hall A, such as Moller polarimeter, e-P energy measurement, Compton Polarimeter, etc.
A note about naming scheme. Prior to Sept 2000 the detector package which is now on the Right Spectrometer was called the Electron Arm. The package on the L-arm was the H-arm. In Sept 2000 the packages were swapped with a crane operation and we realized the old naming scheme made no sense (e.g. both spectrometers can have electrons). The old naming still exists in some places like detector map or some software, though we should be slowly weeding it out. Also, it is quite relevant for older experiments.
CODA DATA FORMAT
The best detailed reference on the CODA format was the old manual for CODA version 1.4, see appendix E. I have saved this postscript file coda1.4.ps. I will give a simplified summary here. The data come in the form of an array of 32-bit words, in the case of event data, or character data, in the case of some of the special events. In general, the first part of the event structure is "header" information, indicating how long the event is, what the event type is, etc. The most important header and other identifying data words are:
| Location in Buffer | Content | Description |
| 1st word | event length |
loop from J=1 to J=evbuffer(1)+1 in Fortran
to access data = evbuffer(J) |
| ishft( evbuffer(2), -16 ) | event type |
Types 1-12 are triggers that don't overlap in
time. Type 14 is any trigger overlap. Other types correspond to prestart, go, end, etc. Types >128 are "special events". |
| evbuffer(4) in event type 17 | run number | The run number, which is normally in the filename also. |
| evbuffer(5) | event number | The event number, uniquely assigned by CODA |
After the header, the data are organized in banks corresponding to each ROC. A ROC is a "readout controller", and it means a crate of fastbus or VME. We have four ROCs in our spectrometer DAQ event stream: 1) ROC1 = fastbus on E-arm (after 9/2000 it is called R-arm); 2) ROC2 = fastbus on H-arm (L-arm) for VDCs and scintillators; 3) ROC3 = fastbus on H-arm for FPP; 4) ROC14 = VME crate for BPM and raster data. In some periods there were more ROCs (13, 15). The beginning of each ROC bank contains the length of the bank and an identifying tag to indicate which ROC number. A bit of Fortran code which illustrates how to crawl through the banks is shown below.
do 150 i = 1,MAXROC
c n1 = pointer to first word of ROC
if(i.eq.1) then
n1 = evbuffer(3) + 4
else
n1 = n1roc(irn(i-1))+lenroc(irn(i-1))+1
endif
c irn = ROC number obtained from CODA
irn(i) = ishft(iand(evbuffer(n1+1),'1f0000'x),-16)
if(irn(i).le.0.or.irn(i).gt.maxroc) then
write(6,*) 'fatal error in decoding'
stop
endif
c Store pointers and lengths
if(i.eq.1) then
n1roc(irn(1)) = n1
lenroc(irn(1)) = evbuffer(n1)
lentot = n1 + evbuffer(n1)
else
n1roc(irn(i)) = n1roc(irn(i-1))+lenroc(irn(i-1))+1
lenroc(irn(i)) = evbuffer(n1roc(irn(i)))
lentot = lentot + lenroc(irn(i)) + 1
endif
if(lentot.ge.len-1) goto 200
150 end do
200 numroc = i
Libraries exist for reading and writing CODA data format. The library is in the ESPACE distribution, in the /evio directory. A convenient example code for reading/writing CODA data (written in C) is here: rwcoda.c with a Makefile (one can link to CODA libraries or simply compile evio.c and link it instead).
The data are decoded by ESPACE in the routine event_decoder which is in the module coda.f. This routine requires the detector map. A recent example of the detector map is shown here: detmap.config. This detector map gives the correspondence between the "raw channel number" and the "analysis detector channel". By "raw channel number" I mean the combination of : (ROC number, slot in ROC, channel within the slot). By "analysis detector channel" I mean the channel within a detector as indexed by ESPACE. The detector map is also stored in the CODA file as a special event and can be used by ESPACE by leaving out (or commenting out) the line "set/file/detmap detmap.config" in the ESPACE kumac. The absence of this line causes ESPACE to search the CODA file for the detmap. The presence of this line causes ESPACE to take the detmap specified in the kumac, which then over-rides whatever is written in the CODA file. The intention, of course, is that the detmap in the CODA file is the correct one at the time of the experiment. There have been some reports that ESPACE occassionally (rarely) fails to find the detmap in a CODA file. You can work around this problem by extracting it once and then renaming this file and using it as the the file pointed to in the kumac file. The extracted detmap file is always called detector_map.lis.
To add a new detector to ESPACE, there is a recipe which affects several routines. To learn the recipe, ask me. If the new detector is added this way, it allows for incorporation in the detector map.
Scalers count raw hits on phototubes, as well as important quantities like charge and triggers which are used to normalize the experiment. The scalers are read online using the "xscaler" code, and the meaning of the channels is evident from the scaler configuration file which that code uses. The scaler config file is normally located in the directory from which you must run "xscaler", namely ~adaq/$EXPERIMENT/electron/ scaler (After 9/2000, it is called ~adaq/$EXPERIMENT/right/ directory) and ~adaq/$EXPERIMENT/hadron/scaler (/left/scaler) where $EXPERIMENT is an environment variable like e97111. The meaning of most of the channels usually does not change between experiments, but there are a few scalers that get rearranged. I have tried to write the mapping shown below in such a way that older experiments can figure out their map. Look at the date in the heading of the table. Hint During each experiment the ``scaler.config'' file in the ``xscaler'' directory (mentioned above) is made correct. Therefore, these files should be saved. I will try to remember to put copies in the experiment's MSS directory.
| Bank of 16 Channels | Header | Usage |
| 1st Bank | 0xceb00010 | 1-6: S1-Left, 7-12: S2-Left PMTs |
| 2nd Bank | 0xceb10010 | 1-6: S1-Right, 7-12: S2-Right |
| 3rd Bank | 0xceb20010 | 1-6 S1-L+R, 7-12: S2-L+R (ANDs) |
| 4th Bank | 0xceb30010 | 1-10 Gas Cerenkov PMTs |
| 5th Bank | 0xceb40010 | Normalization Data [NOTE: H-arm (L-arm) copy is more complete] (channel 1=T1, 2=T2, 8=clock, 9=Unser, 10=Downstream(x3)BCM, 11=Downstream(x10)BCM, 14=Upstream(x1)BCM, 16=Downstream(x1)BCM) |
| 6th Bank | 0xceb50010 | 1-13 Aerogel Detector Left PMTs |
| 7th Bank | 0xceb60010 | 1-13 Aerogel Detector Right PMTs |
| 5th Bank | 0xceb40010 | Normalization Data (channel 1=T1, 2=T2, 5=h+, 6=h-,8=clock, 9=Unser, 10=Downstream(x3)BCM, 11=Downstream(x10)BCM, 14=Upstream(x1)BCM, 16=Downstream(x1)BCM) |
| 6th Bank | 0xceb50010 | Helicity Plus Gated Normalization Data [It is a copy of the 5th bank, gated by helicity] |
| 7th Bank | 0xceb60010 | Helicity Minus Gated Normalization Data [It is a copy of the 5th bank, gated by helicity] |
| Bank of 16 Channels | Header | Usage |
| 1st Bank | 0xabc00010 | 1-6: S1-Left, 7-12: S2-Left PMTs |
| 2nd Bank | 0xabc10010 | 1-6: S1-Right, 7-12: S2-Right |
| 3rd Bank | 0xabc20010 | 1-6 S1-L+R, 7-12: S2-L+R (ANDs) |
| 4th Bank | 0xabc30010 | Helicity Minus Gated Normalization Data [It is a copy of the 5th bank, gated by helicity] |
| 5th Bank | 0xabc40010 | Normalization Data (channel 1=T1, 2=T2, 3=T3, 4=T4, 5=T5, 7=Upstream(x3)BCM, 8=clock, 9=Upstream(x10)BCM, 11=Downstream(x3)BCM, 12=Downstream(x10)BCM, 13=Accepted-Triggers(all), 14=Upstream(x1)BCM, 15=Unser, 16=Downstream(x1)BCM) |
| 6th Bank | 0xabc50010 | Helicity Plus Gated Normalization Data [It is a copy of the 5th bank, gated by helicity] |
| 7th Bank | 0xabc60010 | 1-10 Gas Cerenkov Detector PMTs |
In summer of 1999 we installed new scalers with 32 channels and are using them for the normalization data now. Note that the header contains the number of channels in the first 6 bits, and always has. Decode this way to be backward and forward compatible.
| Now 32 Channels | Header | Usage |
| 4th Bank | 0xabc30020 | Helicity Minus Gated Copy of Normalization data. First 16 channels as before. Channel 17-32 new. |
| 5th Bank | 0xabc40020 | Normalization Data First 16 channels as before (see table above). Channel 17-32 new. 17=charge (Upx10 BCM) gated by DAQ-not-busy, 18=clock gated by not-busy,24=copy of 1024 Hz clock, others empty at present |
| 6th Bank | 0xabc50020 | Helicity Plus Gated Copy of Normalization data. First 16 channels as before. Channel 17-32 new. |
Note for E97111 after detector swap In Sept 2000 the map of channels for triggers was changed because of the detector swap combined with the special need of e97111 to run the trigger supervisor on the right spectrometer. Since we don't have enough cables to carry data between the spectrometers, we could not fanout the trigger data to all the old places. Therefore I added an 8th scaler bank with 32 channels on the right arm, and all trigger data goes there (and goes to some of the old places as well). In addition, we have a pulser to measure electronics deadtime. It is ``and''ed with the trigger on each spectrometer separately. Since July 2000 we also measure the strobe rate on each spectrometer. Map:
L-arm strobe rate. header 0xabc40020, 20th channel
L-arm DT pulser, header 0xabc40020, 22nd channel
L-arm DT.and.trigger, header 0xabc40020, 23rd channel
R-arm strobe rate. header 0xceb40010, 4th channel
For e97111:
R-arm DT pulser, header 0xceb70020, 17th channel (see also table below)
R-arm DT.and.trigger, header 0xceb70020, 18th channel (" ")
For e99007 and beyond:
R-arm DT pulser, header 0xceb80020, 17th channel (see also table below)
R-arm DT.and.trigger, header 0xceb80020, 18th channel (" ")
| 32 Channels | Header | Usage |
| 8th Bank | 0xceb70020 | For e97111 this was the trigger data ungated by helicity, but subsequently (e99007 and beyond) it was trigger data of ceb8 gated by Plus Helicity. |
| 9th Bank | 0xceb80020 | Trigger Data. Channels 1-5 are T1-T5. Channel 8 = 1024 Hz clock. Channel 17 = deadtime pulser, channel 18 = DT pulser.AND.trigger Note: These data dont necessarily go in the previous normalization scalers. See also above for charge info. |
| 10th Bank | 0xceb90020 | Trigger Data from ceb8 gated by Negative helicity. |
The scalers are cleared at the beginning of a run, and read out finally at the end of the run. The timing of the gates, which enables the scalers to count, is such that the end-of-run scaler read is done after gates are turned off, and then after this final readout the gates are re-enabled, so that scalers can count in between runs. The end-of-run scaler data are written automatically to a file on adaqs2 on the adev account: /home/adev/scaler/scaler_history.dat. Routines exist for reading this file and pulling out the particular scaler channel.
Scaler data are also inserted into the data stream as an event type 140 typically every 4 seconds, but not synchronized to any other event. One way to synchronize them is to use the clock which exists in both the scaler data and the physics events. The time interval between scaler updates has varied in history. After July 1999 the capability to read scalers at 30 Hz with zero deadtime has existed. This allows us to run with the helicity delayed if necessary. However, the data are still integrated and inserted at 4 second intervals.
0xfadcb0b6 0xf7510010 0x84f184f2 0x84f684f7 0x84f384f3 0x84f484f4 0x84f484f4 0x84f384f4 0x84f484f5 0x84f484f5The decoding of these data is similar to the Struck 7510 ADC in ROC14 (see section on ROC14). One first looks for the header 0xf751000xx, where 'xx' contains the number of data words. This is the product of the number of channels (always 8) times the number of 16-bit words per channel (here 2). The number of words be could changed, so one should always get it from the header. The data are organized in 32-bit words, i.e. pairs of 16 bits. The highest order 16 bits belong to the first reading, the lowest the next. In each 16-bit word, the lowest 12 bits are the data. So, in this example, each channel gets read twice per event (the readings are separated by 4 microseconds). The data on the first channel is 0x4f1 (first reading), 0x4f2 (2nd). The 2nd channel reads 0x4f6 and 0x4f7, etc. For awhile we had plugged in BPM 8 and 12 here. We dropped the crate during e89044 at their request.
0xfadcb0b4 0x00000002 0xfadc3123 0x0000fc6a 0x000000d4 0x000000c9 0x0000fb66 0x0000ff35 0x0000ff47 0x0000ff3a 0x0000ff52 0x0000ce1a 0x0000cd0a 0x0000cd8d 0x0000ce07 0x0000cd3a 0x0000cc37 0x0000cbdd 0x0000cf43 0xfadc1182 0x000006a2 0x0000096d 0x0000097d 0x00000190 0x00000cd4 0x0000018f 0x00000757 0x0000018b 0x00000001 0xfadd1182 0x0000092b 0x00000946 0x0000091c 0x0000090d 0x0000094c 0x00000952 0x00000980 0x000008d8 0x00000001 0xfca56000 0x00000005 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
NOTE: After Nov 10, 1999 there will be additional data. The dump will look different.
In general for VME data, a device is demarked by a unique header. Six possible devices may, or may not, exist: one ADC model VMIC3123, two ADC model LeCroy1182, one scaler CAEN-V560, and two ADC model STR7510. The STR7510 is new as of Nov 10, 1999, and allows for the ``burst mode'' readout as described below. The headers of these six devices are respectively 0xfadc3123, 0xfadc1182, 0xfadd1182, 0xfca56000, 0xf7510xxx, and 0xf751xxx (where you should assume xxx can be anything, explained later). Following the headers are a number of channels, 16 for VMIC3123, 8 for LeCroy1182, at least 1 and sometimes 16 for the CAEN-V560, and a variable number N for the STR7510 where N is decoded from its lower bits as explained later. One should decode in such a way that any of these devices may be absent, e.g. I may drop ones that are not needed. One should search for the header, then crawl through the bank beyond that header.
The scaler is the easiest: The first channel is a 100 kHz clock, and at present no other channels are used. This clock is a time stamp for the events. (Note: After April 6, 2000 the scaler has an extra word after the header. This word denotes the number of words that follow. E.g. if the word is 2, it means two channels of scaler follow.)
Prior to Nov 10, 1999 the BPM and raster data have been fanned out to both a VMIC3123 and LeCroy1182. After Nov 10, 1999 the STR7510 ADC replaces the VMIC3123, and the signals are still read redundantly by the LeCroy1182. The BPM data are raw signals on antennas that are at 45 degrees to floor. The raster signal is proportinonal to the instantaneous current in the raster. The raster derivative is the sign of the slope of the raster current. It is high or low.
An event pulse is put into one channel of the LeCroy1182 ADC. It is a signal that increments in sawtooth fashion for each event. This signal goes to one ADC in each crate (also fastbus), to verify synchronization.
A note about some old history on the LeCroy ADC with header fadd1182. It existed in the datastream from about Sept 1, 1998 until now with the exception of the time period Nov 20, 1998 - Jan 1, 1999. At 16:00 Nov 20, 1998 we had to take it out of the ROC14 to move it to ROC15 because we had only two such ADCs. LeCroy finally delivered, so that now both ROC14 and ROC15 have two 1182 ADCs, enough to make all twelve signals read redunadantly by the two kinds of ADCs. This note only affects experiment e94010.
Why we read the same data with two kinds of ADCs: The VMIC3123 had the advantage that it takes a signal +/- 5V straight from the electronics, and this ADC had been recommended by some engineers in ACC division. But it shows a problem that occassionally (quite rarely) it gets out of synch !! The LeCroy ADC apparently never gets out of synch, but we've sometimes had trouble adjusting the signal so that under no circumstances does it saturate. After Jan 1, 1999 the LeCroys don't saturate (I think). After Nov 10, 1999 we use the STR7510 instead of VMIC3123. The mapping of the channels for ROC14 is given below.
------------- Beam Position Monitors 3A and 3B -------------------
BPM Data VMIC3123 channel LeCroy 1182 Channel STR7510 Channel
antenna raw data (new, Nov 10, 1999)
(X1,Y1 at BPM3A
X2,Y2 at BPM3B)
(header) (header)
X1+ 9 (0xfadd1182) 1 (0xf7510xxx) 1
X1- 10 " " 2 " " 2
Y1+,Y1-,X2+,X2-,Y2+,Y2- 11-16 " " 3-8 " " 3-8
Event Pulse Not plugged in (0xfadc1182) 6 Not plugged in
------------------ Raster Data ----------------------------------
Raster Data VMIC3123 channel LeCroy 1182 Chan STR7510 Chan
X-current 1 (0xfadc1182) 7 (0xf7511xxx) 1
Y-current 4 " " 1 " " 2
X-derivative 2 " " 2 " " 3
Y-derivative 3 " " 3 " " 4
From Nov 20, 1998 until Jan 1, 1999 the fadd1182 ADC was relocated
in ROC15 crate. So, during this time we move BPM#2 into fadc1182
in the following channels (and BPM#1 was not read out):
BPM#2 Data VMIC3123 channel LeCroy 1182 Channel
(unchanged)
X2+ 13 4
X2- 14 5
Y2+ 15 6
Y2- 16 8
The channel ordering is admittedly wierd. But once it was done
(by a nameless person), it was less confusing to leave it.
Here is information about the ``burst mode'' readout of the
BPM and Raster which exists after Nov 10, 1999. The STR7510 ADC
allows to read on each event N pulses separated in time by
T microseconds. Typically we choose to read N=6 pulses at
T=4 microsec intervals which gives a snapshot of both the
amplitude and phase of the signals. For tests N and T may
be arranged quite differently (see Bob Michaels for details).
Below is a bit of C++ code showing the decoding.
header14str[0] = 0xf7510000;
header14str[1] = 0xf7511000;
// ...imbedded in a loop over event buffer:
for (k=0; k< NUM7510; k++) {
if((rdata.evbuffer[ipt]&0xfffff000)==header14str[k]) {
itemp = (rdata.evbuffer[ipt]&0xfff)/8;
// nhit=number of hits, will always be EVEN, typically 6.
nhit = (itemp < HITDEPTH) ? itemp : HITDEPTH; //HITDEPTH=array size.
// loop over 8 channels
for(m=0; m<8; m++) {
khit = 0;
for(j=0; j< nhit/2; j++) {
ipt++;
itemp = rdata.evbuffer[ipt];
// k=index of ADC unit; m=index of channel inside unit; khit=hit index.
str7510_raw[k][m][khit++]=(int)((itemp&0x0fff0000)>>16);
str7510_raw[k][m][khit++]=(int)(itemp&0xfff);
}
}
}
}
Here is a typical raw data dump of ROC15 (post Jan 1, 1999)
0xfadcb0b5 0x00000002 0x0000ff80 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xfadd3123 0x0000cba2 0x0000cbbb 0x0000cbd8 0x0000cc1e 0x0000cc05 0x0000cc17 0x0000cc1d 0x0000ce99 0x0000ff24 0x0000ff0b 0x0000ffee 0x000000ad 0x0000ff34 0x0000ff45 0x0000ff70 0x0000ff3e 0xfade1182 0x00000120 0x00000124 0x00000132 0x00000122 0x000009f1 0x000009ea 0x000009ee 0x00000988 0xfadf1182 0x00000a2e 0x00000a0e 0x00000977 0x000009b1 0x00000116 0x0000010b 0x0000011e 0x00000116
The detector map for ROC15 is as follows:
Header: 0xfadd3123 (model 3123 ADC).
Channel Data
1-8 X1+,X1-,Y1+,Y1-,X2+,X2-,Y2+,Y2-
(see notation for ROC14)
9 Raster X-current
10 Raster Y-current
11 X-derivative
12 Y-derivative
After Nov 20, 1998 switch to 2-TS setup, there will be a LeCroy 1182 ADC in this crate, for redundancy. Ideally we'd have at least 12 channels, but only 8 were available as of Nov 20, 1998 This is important becaues the VMIC ADC sometimes exhibits synch problems.
Header: 0xfade1182
Channel Data
1 X-raster
2 Y-raster
3 X-derivative
4 Y-derivative
5,6,7,8 respectively X2+,X2-,Y2+,Y2-
After Jan 1, 1999 we were able to add a second LeCroy 1182 ADC. We then add the following channels (while the above remains the same)
Header: 0xfadf1182 Channel Data 1,2,3,4 respectively X1+,X1-,Y1+,Y1-
Data from various EPICS databases are periodically inserted into the datastream, as well as placed in text files that are saved at the start-of-run and end-of-run. These text files are stored as, for example, /home/adev/epics/runfiles/Start_of_Run_1047.epics, and End_of_Run_1047.epics where 1047 is the run number. These data are also written to the electronics log book HALOG. Each experiment is permitted to modify the script that controls this (ask me) but please let only ONE experienced, qualifed person modify the script (and it is probably good enough, too). Recently Mark Jones has been maintaining this script. Incidentally, no other script dealing with DAQ should ever be modified.
The data which are inserted periodically include the beam current, position monitors, magnetic fields in our spectrometers, as well as some beamline elements, the beam energy, collimator positions, and several other things. Approximately every 30 seconds, a long list of information from the accelerator and from Hall A are inserted as event type 131. Approximately every 5 seconds a shorter list of information, for which rapid updates are desirable, are inserted as event type 131. The shorter list contain the BPMs near the target and the charge monitor data. The synchronization of the data relative to our datastream is good to perhaps 1 second.
Here is an example of the rapidly updated EPICS data which are inserted every approximately 5 seconds :
Tue Aug 25 12:59:43 EDT 1998 IPM1H03A.XPOS 0.352823 IPM1H03A.YPOS 0.430828 IPM1H03B.XPOS -0.130145 IPM1H03B.YPOS -0.48034 hac_bcm_average 5.29884 hac_bcm_dvm1_current 5.30776 hac_bcm_dvm2_current 5.28991 hac_unser_current 5.30134
Here is a link showing more fully some examples of EPICS data, though some details have changed since that information was posted. One can see that the data are in the form of a series of lines of character data, where each line is of the form "keyword data-value", where keyword is a descriptive word (actually the name of the EPICS variable) and data-value is its numerical value.
Event types less than 15 are physics triggers of various types. There are also prestart, go, and end "events". Event types greater than 127 are "special events", which are not really "events" and not particularly "special" either -- they are data inserted into the CODA datastream which are useful to record with the event data. The following special events exist:
| Event Type | Format | Description |
| 131 | character | EPICS Data (see section above) |
| 133 | character | Prescale Factors (decoded by ESPACE) |
| 135 | character | Detector Map (optionally used by ESPACE) |
| 136 | character | Trigger Setup (read it and see) |
| 140 | integer | Scaler Data (each spectrometer every few seconds) |