ComptonDaqDev
Here are some notes on the Compton DAQ Development.
Maybe should go into a logbook.
Start date Sept 26, 2012
Bob Michaels, Kalyan Allada
Will work on adev account.
Want to use CODA 2.6. Save the old tcshrc file cp .tcshrc tcshrc_22feb2012
VME cpu bbvme1 presently on bbps1 2002
FADC library for VME /adaqfs/halla/adev/fadc/v1.2
crl is /adaqfs/home/adev/fadc/v1.2/fadc_list.crl
Decoding of FADC /adaqfs/halla/adev/fadc/ComptonAnalyzer
Attempts to make CODA 2.6 work on adev
1. Adjusted .tcshrc, adding things at top (comments there).
2. Create coda26 directory. Put $COOL_HOME there.
3. Minor problems. Had to start msqld by hand (/etc/rc.d/rc5.d/S82msqld_s on adaql2). Had to run cedit or dbedit on adaql1, not adaq1, b/c on adaq1 it complains cedit: error while loading shared libraries: libXm.so.3: (adaq1 has libXm.so.4)
4. Minor problems (part II). Needed links on adaq1, else CRLs don't compile. in /usr/local: ln -s /adaqfs/coda/vxworks-linux gcc-vxworks and ln -s /adaqfs/halla/apar/vxworks/5.5/gnu_vxworks gnu-ppc
5. Minor problems (part III). Can't run rcgui on adaq1 b/c missing libXtst.so.6. I guess it's looking in the wrong place since it's here: /usr/lib64/libXtst.so.6. Punt on adaq1. Use adaql1 (with the "l") for now.
6. startcoda on adev was a link to startcoda_2.5. Change this to startcoda_2.6
7. Bootscript on bbvme1 changed from /adaqfs/home/adev/bbite/vxworks/bbvme1_coda25.boot to /adaqfs/home/adev/vxworks/bbvme1_coda26.boot
8. Adjusted hardware address of TIR and FADC in CRL.
9. Started all the DAQ components "by hand" (startcoda not working yet). A working config is $SESSION=Compton, Configuration = Compton2. A seemingly ok data file is /adaql1/data4/AComp_1012.dat.0
10. "startcoda" and "kcoda" are now working on adev account. A working configuration is "Compton2".
to be continued ....
Date : 28th Sept 2012 (K.Allada)
1. As a first step trying to decode fadc data using /adaqfs/halla/adev/fadc/fDecode which prints data on screen.
2. The 'fadc_decode' prints data with any slot and crate number as an argument ...this is not expected.
3. I tried to change some parameters in the crl (such as PTW and PL) and the decoder shows correct changes made to PTW, at least that works!(run 1019)
4. Now I increased the delay of the trig to TI so as to catch the pulse (it wasn't sufficient), but the decoder still see no signal, only pedestal. (run 1020)
5. Added some more delay(~200ns) between trigger and signal, changed PL=100(400ns) and PTW=80(320ns) in the crl code. Now I can see the signal from the decoder in fDecode!! See halog (run 1028)
6. For now, this is the only working decoder (/adaqfs/halla/adev/fadc/fDecode)... more later.
Date : 1 Oct 2012 (R. Michaels)
1. Got my own version of fDecode to compile and run in ~adev/bob. Spent about an hour figuring out where the heck the slot number comes from (I did). Slot 5 for now.
2. Latency 300 nsec. Trigger window 260 nsec. I learned how to adjust this. Runs 1040 - 1045 are ok. I can see the data, and it moves to another channel when I move the input cable, but there are some strange things: (a) For data in chan 15, I see many times a repitition of the same ADC data, it's not physical. Meanwhile for chan 16, the data looks normal (monotonic, looks like a pulse). And (b) When I do run #N the first few events look like run N-1 conditions, as if the ADC is not cleared. Obviously, some work to do here.
3. Neither fDecode nor decoder_new compiles on adaq1. Instead must use adaql* (with the "L"). The CODA-related problems for adaq1 were allegedly fixed, so we may try running CODA there again sometime. I'm more interested in the FADC for now, though.
Date : 2 Oct 2012 (K. Allada)
1. I replaced the Lecroy linear Fan-in/Fan-out module with another one. This was most likely causing flat line on the input pulser signal, now it is gone.
2. Input pulse is ~400mV and the disc. threshold is set to ~300mV. On the scope everything looks clean, no jumping signals and flat-lines.
Date : 3 Oct 2012 (K. Allada)
1. I got Alexander's version of the decoder working and wrote my own script to display fadc channels on event-by-event basis. The new decoder is located at /adaqfs/home/adev/fadc/fadc_decoder. See README file to run the event display.
2. The Lecroy 428F unit in the NIM crate is a modified version, with two of the four outputs (B and D) capacitively coupled, hence stable. O/P A and C are unstable. Therefore always use o/p B and D for testing. (see diagram on the unit)
Date : 8 Oct 2012 (R. Michaels)
1. Decoder bug: If I turned on pedestal suppression (aka faSetThreshold), it exposed a bug in fadc_decoder, specifically that code looked for the channel to change, which it does not if only one channel has data. I fixed this in my private version (~/bob). My fix is somewhat awkward, so it may need to be checked.
2. I noticed that the reasons why an input signal appeared to be necessary to have triggers was some kind of grounding issue. If I just connect to ground the "ground" part of the LEMO input for data inputs on the FADC, the triggers occur. It is not necessary to have data, only this ground connection is relevant. If this connection is absent, the DAQ is stuck: no trigges. Very odd. I have asked the DAQ group why.
3. I added two scalers (Struck model 3800) to the crate. One for LEMO inputs and another for ECL inputs. Right now there is a "scread" (standalone code to read) in ~adev/scaler/compton/scread. Go there and type "scread" to see the data from the server. And "scread -c" to clear. It works. I have not yet added these scalers to the CRL, but it's easy -- I might do it tomorrow -- but it should be easy to remove (with a flag) since it will affect the ultimate speed of the readout. I suggest putting the readout at the end of the event structure. It will affect the decoder, too. The decoder should sense if there is a scaler or not.
4. I think what we want is a setup with a pulser (ideally random but can start with a fixed rate), use pedestal suppression on all but one channel, put the signal into that one channel, and observe how many FADC events we see versus how many triggers in in the scaler -- essentially measure the deadtime. We should map this out versus rate and see how limited the system is at present.
5. Task list: a) Try the internal trigger (HITSUM, or whatever) b) Need a random source of triggers (PMT in a box with controllable light leak) c) Question about the grounding (note 1 above). Other question about how to ignore a channel, i.e. no pedestal suppression but don't read out some channels. d) Measure deadtime versus rate. Can use scaler server to read at beginning and end of a long run.
Date : 10 Oct 2012 (R. Michaels)
Today's progress
a) Found out we can't use HITSUM internal trigger on this version (v2) of board; will get a new firmware someday.
b) I learned how to disable channels. At the moment, all but channel 5 (i.e. 5th) is disabled. Warning !
c) Found out we cannot try 2eSST mode, which should speed up VME by a factor of 2 to 4, b/c we have a 5100 board (need a 6100 board and the Tempe library)
d) Implemented a deadtime measurement on scope -- read out busy. Also, studied observed rate versus input rate.
Conditions: 5100 board, 100 Mbit ethernet, old TI (not the new one), no event blocking (b/c old TI), 1 channel readout, no pedestal suppression, but other channels disabled. The busy occurs 4 usec after the trigger. The VME readout deadtime is usually 25 usec, but fluctuates sometimes to 40 usec. This implies a max rate of order 25 to 40 kHz, which was observed (see below).
Data for a fixed-frequency pulser (later, it would be interesting to use a random trigger)
Input rate .... Observed rate .... deadtime (kHz) ....... (kHz) 10.2 .......... 9.61 ............ 5.8 % 13.5 .......... 12.1 ............ 10.3 % 34.0 .......... 16.3 ............ 52 % 21.5 .......... 17.6 ........... 18.1 % 35.2 .......... 16.1 ........... 54 % 44.0 .......... 17.8 ........... 60 %
Looking forward to this improving with new cpu, new TI, event blocking, 1 Gbit ethernet, 2eSST, and whatever else we can think of.
Date : 11 Oct 2012 (A. Camsonne)
Lending one of HRS intel VME CPU to test with FADC Board is VXB601, MAC address is 00:20:38:04:6D:12 and hostname is hallasfi1.
To compile, first source sethallasfi1.sh to set the environment variables. Then you can compile in the linuxvme directory.
The FADC library is located in fadclib and the standard readout list is fadc_list.c
The ComptonIntel configuration was created. The Intel CPU is setup as ROC9 To start the ROC if it is not start execute the startroc9 script at the root directory.
I copied a version of the SIS scaler library for the linux VME CPU in /root/linuxvme/SISScaler this needs to be setup for this scaler ( VME addresses )
K.Allada: Things to check with intel CPU:
1) Repeat the the DT test done earlier. For this, we need scread to work on the hallasfi1
2) To compile /root/linuxvme/SISScaler/fadc_list.c, do "make rol".
Date : 12 Oct 2012 (R. Michaels)
Today:
1) I tried out the new Intel VME cpu (see Oct 11) and it works nicely. I learned about this setup and checked the data from the FADC.
2) I ported my scaler readout to the Intel VME cpu. After some effort to get Makefiles right, and reading https://coda.jlab.org/wiki/index.php/VxWorks_to_Linux_HOWTO I was able to get it to work, i.e. it can initialize, read, and clear scalers. Note, it's not a client/server anymore; at this point the code just runs on the Intel board. There will still be some benefit to having client/servers, and I'll work on that later.
3) Another very cool thing is that I demonstrated that I can use the serial port to talk to the HV main frame using primtive RS232 commands. This is nice, because it means that anywhere we have these cpus we can talk to HV crates.
Date : 15 Oct 2012 (R. Michaels)
Scaler readout using "scread" on hallasfi1 (Intell cpu): Type "scread" to read the two scalers in that VME crate. "scread -c" to clear. And "getrate" will make an printout of the rates (what's printed are values and you have to divide by 10 sec.)
I noticed that with the Intel VME board, there is a clear difference in readout speed when using 2eSST.
vmeDmaConfig(2,5,1) -- optimal. 26 usec for 1 channel
vmeDmaConfig(2,2,1) -- D32 is 4 times slower (100 usec).
In the present setup, a pulser provides the trigger and the data. The signal is discrimated and send to the TI board; the L1A is then sent to the FADC timing board, which feeds the FADC its trigger. The signal also goes to the data input on the FADC.
What is a bit wierd is that there is a 25 usec delay between the start of the readout (i.e. busy) and the trigger sent to the TI. This effectively adds 25 usec of deadtime to the system. I'd like to understand and possibly eliminate this. If we can, the system has a 26 usec DT in 2eSST, which implies a 40 kHz rate capability.
Date : 16 Oct 2012 (K. Allada)
1. Found that the data in the run I just took have a bunch of 0xf0000000 in the file. Compiled the readout list and took another run, still same ... something wrong.
2. Yesterday B.Moffit suggested me to use readout list version2 in /root/linuxvme/fadcV2/rol/fadc_list.c. So I changed the ComptonIntel config. to point to fadc_list.so in this location. Also, made made sure old and new lists are same.
Comment from Alexandre : we have an old FADC so fadcV2 will fail
3. Also /root/linuxvme/lib/libfadc.so is pointing to fadcV2/
4. I see 26usec DT and data contains only 0xf0000000. Not sure whats wrong.
5. Went back to old 5100 CPU and ran Compton2 config. Everything looks good in the data (no 0xf0000000). Also, I noticed that the delay between busy and the trigger sent to TI is 5-8usec (rather than 25 usec that is seen with Intel CPU).
6. ( Alexandre ) Intel CPU did not reboot, it did not pick up the IP address. I restarted the DHCP server on adaql2 : /etc/rc.d/init.d/dhcpd restart and the CPU booted fine.
Date : 17 Oct 2012 Theory of where we're going -- Bob Michaels
We want to demonstrate 1 MHz capability, or something approaching that, for the Compton photon readout.
The setup we have: 1 channel is read out from the FADC, the others are disabled. This 1 channel will be the Compton photon detector data.
Other parameters and discussion:
1. We have a window of approx 300 nsec, so that's 75 words at 250 MHz, plus about 5 for headers and trailers, etc, so about 80 words per event
2. The picture of the deadtime: A trigger comes into the TI board and it takes a time "Tz1" to interrupt the board and a time "Tz2" to read out. At present, with the Intel board, Tz1 = 25 usec and Tz2 = 26 usec, approximately. No way to reach 1 MHz !
3. In polling mode, Tz1 should be about 3 usec, I'm told. We have not demonstrated this yet in our setup, so there must be a mistake.
4. The overhead to read out is about 20 usec (Tz2), but not because of DMA transfer (which is 40 nsec per word, so 3.2 usec for 80 words). Instead Tz2 is dominated by the overhead of setting up, on the cpu, where to write the data. This setup time would get buffered away if we used event-blocking. Event-blocking requires the new TI board which we are waiting for.
5. In event-blocking mode, we can buffer the events on the FADC and read them all at once. The buffer size is 4 Mbytes. Given 80 words or 20 bytes per event (see 1), we could store 200K events in the FADC -- a lot !
6. The DMA transfer speed in 2eSST is D = 40 nsec/word * number of words/event = 3.2 usec for our present setup (see 4).
7. Suppose we buffer 10K events. Then the readout time per event is (Tz1/10K) + (20 usec/10K) + D, where the first time is the polling delay, the 2nd is the setup time to prepare where to write on the cpu, and the D = 3.2 usec is the DMA transfer of 20 bytes per event. Clearly, the 3.2 usec dominates. This implies 313 kHz rate limit.
8. Ultimate speed limit ? If we get greedy and reduce the time window for pulses to 50 nsec (assuming the pulses fit) this could become D = 40 nsec * ((50 nsec * 250 MHz + 5)) = 0.7 usec, so deadtime-free at 1.4 MHz theoretical rate limit. With pipelining, the deadtime would be close to zero at this rate.
Please check this.
Date : 17 Oct 2012 (K.Allada)
1. fadc readout list now pointing back to the one in /root/linuxvme/fadclib. I did "make" and that linked all the libraries back to this folder. This fixed the bad data (0xf0000000) issue.
