Oct 2012
Date : 1 Oct 2012 (R. Michaels)
1. Got my own version of fDecode to compile and run in ~adev/bob. Spent about an hour figuring out where the heck the slot number comes from (I did). Slot 5 for now.
2. Latency 300 nsec. Trigger window 260 nsec. I learned how to adjust this. Runs 1040 - 1045 are ok. I can see the data, and it moves to another channel when I move the input cable, but there are some strange things: (a) For data in chan 15, I see many times a repitition of the same ADC data, it's not physical. Meanwhile for chan 16, the data looks normal (monotonic, looks like a pulse). And (b) When I do run #N the first few events look like run N-1 conditions, as if the ADC is not cleared. Obviously, some work to do here.
3. Neither fDecode nor decoder_new compiles on adaq1. Instead must use adaql* (with the "L"). The CODA-related problems for adaq1 were allegedly fixed, so we may try running CODA there again sometime. I'm more interested in the FADC for now, though.
Date : 2 Oct 2012 (K. Allada)
1. I replaced the Lecroy linear Fan-in/Fan-out module with another one. This was most likely causing flat line on the input pulser signal, now it is gone.
2. Input pulse is ~400mV and the disc. threshold is set to ~300mV. On the scope everything looks clean, no jumping signals and flat-lines.
Date : 3 Oct 2012 (K. Allada)
1. I got Alexander's version of the decoder working and wrote my own script to display fadc channels on event-by-event basis. The new decoder is located at /adaqfs/home/adev/fadc/fadc_decoder. See README file to run the event display.
2. The Lecroy 428F unit in the NIM crate is a modified version, with two of the four outputs (B and D) capacitively coupled, hence stable. O/P A and C are unstable. Therefore always use o/p B and D for testing. (see diagram on the unit)
Date : 8 Oct 2012 (R. Michaels)
1. Decoder bug: If I turned on pedestal suppression (aka faSetThreshold), it exposed a bug in fadc_decoder, specifically that code looked for the channel to change, which it does not if only one channel has data. I fixed this in my private version (~/bob). My fix is somewhat awkward, so it may need to be checked.
2. I noticed that the reasons why an input signal appeared to be necessary to have triggers was some kind of grounding issue. If I just connect to ground the "ground" part of the LEMO input for data inputs on the FADC, the triggers occur. It is not necessary to have data, only this ground connection is relevant. If this connection is absent, the DAQ is stuck: no trigges. Very odd. I have asked the DAQ group why.
3. I added two scalers (Struck model 3800) to the crate. One for LEMO inputs and another for ECL inputs. Right now there is a "scread" (standalone code to read) in ~adev/scaler/compton/scread. Go there and type "scread" to see the data from the server. And "scread -c" to clear. It works. I have not yet added these scalers to the CRL, but it's easy -- I might do it tomorrow -- but it should be easy to remove (with a flag) since it will affect the ultimate speed of the readout. I suggest putting the readout at the end of the event structure. It will affect the decoder, too. The decoder should sense if there is a scaler or not.
4. I think what we want is a setup with a pulser (ideally random but can start with a fixed rate), use pedestal suppression on all but one channel, put the signal into that one channel, and observe how many FADC events we see versus how many triggers in in the scaler -- essentially measure the deadtime. We should map this out versus rate and see how limited the system is at present.
5. Task list: a) Try the internal trigger (HITSUM, or whatever) b) Need a random source of triggers (PMT in a box with controllable light leak) c) Question about the grounding (note 1 above). Other question about how to ignore a channel, i.e. no pedestal suppression but don't read out some channels. d) Measure deadtime versus rate. Can use scaler server to read at beginning and end of a long run.
Date : 10 Oct 2012 (R. Michaels)
Today's progress
a) Found out we can't use HITSUM internal trigger on this version (v2) of board; will get a new firmware someday.
b) I learned how to disable channels. At the moment, all but channel 5 (i.e. 5th) is disabled. Warning !
c) Found out we cannot try 2eSST mode, which should speed up VME by a factor of 2 to 4, b/c we have a 5100 board (need a 6100 board and the Tempe library)
d) Implemented a deadtime measurement on scope -- read out busy. Also, studied observed rate versus input rate.
Conditions: 5100 board, 100 Mbit ethernet, old TI (not the new one), no event blocking (b/c old TI), 1 channel readout, no pedestal suppression, but other channels disabled. The busy occurs 4 usec after the trigger. The VME readout deadtime is usually 25 usec, but fluctuates sometimes to 40 usec. This implies a max rate of order 25 to 40 kHz, which was observed (see below).
Data for a fixed-frequency pulser (later, it would be interesting to use a random trigger)
Input rate .... Observed rate .... deadtime (kHz) ....... (kHz) 10.2 .......... 9.61 ............ 5.8 % 13.5 .......... 12.1 ............ 10.3 % 34.0 .......... 16.3 ............ 52 % 21.5 .......... 17.6 ........... 18.1 % 35.2 .......... 16.1 ........... 54 % 44.0 .......... 17.8 ........... 60 %
Looking forward to this improving with new cpu, new TI, event blocking, 1 Gbit ethernet, 2eSST, and whatever else we can think of.
Date : 11 Oct 2012 (A. Camsonne)
Lending one of HRS intel VME CPU to test with FADC Board is VXB601, MAC address is 00:20:38:04:6D:12 and hostname is hallasfi1.
To compile, first source sethallasfi1.sh to set the environment variables. Then you can compile in the linuxvme directory.
The FADC library is located in fadclib and the standard readout list is fadc_list.c
The ComptonIntel configuration was created. The Intel CPU is setup as ROC9 To start the ROC if it is not start execute the startroc9 script at the root directory.
I copied a version of the SIS scaler library for the linux VME CPU in /root/linuxvme/SISScaler this needs to be setup for this scaler ( VME addresses )
K.Allada: Things to check with intel CPU:
1) Repeat the the DT test done earlier. For this, we need scread to work on the hallasfi1
2) To compile /root/linuxvme/SISScaler/fadc_list.c, do "make rol".
Date : 12 Oct 2012 (R. Michaels)
Today:
1) I tried out the new Intel VME cpu (see Oct 11) and it works nicely. I learned about this setup and checked the data from the FADC.
2) I ported my scaler readout to the Intel VME cpu. After some effort to get Makefiles right, and reading https://coda.jlab.org/wiki/index.php/VxWorks_to_Linux_HOWTO I was able to get it to work, i.e. it can initialize, read, and clear scalers. Note, it's not a client/server anymore; at this point the code just runs on the Intel board. There will still be some benefit to having client/servers, and I'll work on that later.
3) Another very cool thing is that I demonstrated that I can use the serial port to talk to the HV main frame using primtive RS232 commands. This is nice, because it means that anywhere we have these cpus we can talk to HV crates.
Date : 15 Oct 2012 (R. Michaels)
Scaler readout using "scread" on hallasfi1 (Intell cpu): Type "scread" to read the two scalers in that VME crate. "scread -c" to clear. And "getrate" will make an printout of the rates (what's printed are values and you have to divide by 10 sec.)
I noticed that with the Intel VME board, there is a clear difference in readout speed when using 2eSST.
vmeDmaConfig(2,5,1) -- optimal. 26 usec for 1 channel
vmeDmaConfig(2,2,1) -- D32 is 4 times slower (100 usec).
In the present setup, a pulser provides the trigger and the data. The signal is discrimated and send to the TI board; the L1A is then sent to the FADC timing board, which feeds the FADC its trigger. The signal also goes to the data input on the FADC.
What is a bit wierd is that there is a 25 usec delay between the start of the readout (i.e. busy) and the trigger sent to the TI. This effectively adds 25 usec of deadtime to the system. I'd like to understand and possibly eliminate this. If we can, the system has a 26 usec DT in 2eSST, which implies a 40 kHz rate capability.
Date : 16 Oct 2012 (K. Allada)
1. Found that the data in the run I just took have a bunch of 0xf0000000 in the file. Compiled the readout list and took another run, still same ... something wrong.
2. Yesterday B.Moffit suggested me to use readout list version2 in /root/linuxvme/fadcV2/rol/fadc_list.c. So I changed the ComptonIntel config. to point to fadc_list.so in this location. Also, made made sure old and new lists are same.
Comment from Alexandre : we have an old FADC so fadcV2 will fail
3. Also /root/linuxvme/lib/libfadc.so is pointing to fadcV2/
4. I see 26usec DT and data contains only 0xf0000000. Not sure whats wrong.
5. Went back to old 5100 CPU and ran Compton2 config. Everything looks good in the data (no 0xf0000000). Also, I noticed that the delay between busy and the trigger sent to TI is 5-8usec (rather than 25 usec that is seen with Intel CPU).
6. ( Alexandre ) Intel CPU did not reboot, it did not pick up the IP address. I restarted the DHCP server on adaql2 : /etc/rc.d/init.d/dhcpd restart and the CPU booted fine.
Date : 17 Oct 2012 Theory of where we're going -- Bob Michaels
We want to demonstrate 1 MHz capability, or something approaching that, for the Compton photon readout.
The setup we have: 1 channel is read out from the FADC, the others are disabled. This 1 channel will be the Compton photon detector data.
Other parameters and discussion:
1. We have a window of approx 300 nsec, so that's 75 words at 250 MHz, plus about 5 for headers and trailers, etc, so about 80 words per event
2. The picture of the deadtime: A trigger comes into the TI board and it takes a time "Tz1" to interrupt the board and a time "Tz2" to read out. At present, with the Intel board, Tz1 = 25 usec and Tz2 = 26 usec, approximately. No way to reach 1 MHz !
3. In polling mode, Tz1 should be about 3 usec, I'm told. We have not demonstrated this yet in our setup, so there must be a mistake.
4. The overhead to read out is about 20 usec (Tz2), but not because of DMA transfer (which is 40 nsec per word, so 3.2 usec for 80 words). Instead Tz2 is dominated by the overhead of setting up, on the cpu, where to write the data. This setup time would get buffered away if we used event-blocking. Event-blocking requires the new TI board which we are waiting for.
5. In event-blocking mode, we can buffer the events on the FADC and read them all at once. The buffer size is 4 Mbytes. Given 80 words or 320 bytes per event (see 1), we could store 12.5 K events in the FADC -- a lot !
6. The DMA transfer speed in 2eSST is D = 40 nsec/word * number of words/event = 3.2 usec for our present setup (see 4).
7. Suppose we buffer 10K events. Then the readout time per event is (Tz1/10K) + (20 usec/10K) + D, where the first time is the polling delay, the 2nd is the setup time to prepare where to write on the cpu, and the D = 3.2 usec is the DMA transfer of 20 bytes per event. Clearly, the 3.2 usec dominates. This implies 313 kHz rate limit.
8. Ultimate speed limit ? If we get greedy and reduce the time window for pulses to 50 nsec (assuming the pulses fit) this could become D = 40 nsec * ((50 nsec * 250 MHz + 5)) = 0.7 usec, so deadtime-free at 1.4 MHz theoretical rate limit. With pipelining, the deadtime would be close to zero at this rate.
Please check this.
(K.Allada)
1. fadc readout list now pointing back to the one in /root/linuxvme/fadclib. I did "make" and that linked all the libraries back to this folder. This fixed the bad data (0xf0000000) issue.
2. I noticed that when the input rate is greater than ~9 kHz then coda hangs and stops taking data. Bob saw this issue earlier, and it is happening due to running scread when coda is running. Need to fix scread.
Date : 19 Oct 2012 (K.Allada)
1. Came down to the Hall with B.Moffit to take a look at our setup. Bryan noticed that "tirIntInit()" in the download routine is setting the TIR mode to "interrupt on trigger" even if we set the mode to polling at the beginning. The already included linuxvme_list.c will initialize the tir, so there is no need to initialize it again in the download routine.
2. After removing the tir initialization in the download the delay between the tir busy (BSY) and the readout from one of the TIR o/p (readout time) is about 3usec (which is what we expect). Now we are limited by the 26usec wide of readout time which can be overcome by using blocking (need new TI).
3. The maximum rate we could get is about ~10-12KHz at the moment, which is still less than the expected ~40KHz. Part of the reason why we get less max. rate is due the limitation in writing to the disk (disk speed). When we ask coda not to write to disk then we get max rate of about 20-22KHz. So disk speed seems to be a limitation!
4. Bryan pointed me to one of his earlier studies done to show the improvement in speed when using blocking mode using the new TI. See here: https://halldweb1.jlab.org/wiki/images/f/fa/Slide20.jpg. You can see that when blocklevel=1(something like our setup) the speed is limited by the data rate(flat).
5. We are at the moment ready for a new TI and push the rate as much as possible.
Date : 21 Oct 2012 (K.Allada)
1. Disk write speed on adaql1 = 392 MB/s , on adaq1 = 1.1 GB/s
2. Use "dd" utility to find these numbers (Eg: run command "dd if=/dev/zero of=/tmp/output.img bs=8k count=256k", and remove the /tmp/output.img after done).
Date : 22 Oct 2012 (R. Michaels)
To simulate buffering, which would reduce DT, I tried to skip events (a number = BUFFSIZE) in the trigger routine, i.e.read out every BUFFSIZE triggers. What I expected was to see BUFFSIZE-1 empty events (this was observed) followed by 1 event with BUFFSIZE*W bytes, where W is proportional to the window size. Although I did see that W was proportional to window size, I only got one window's worth of data, as if the FADC is cleared each time it's read out. I posed the question to B. Moffit.
B. Moffit's answer was that we may be using the wrong block level. We were using 1. I tried block level BUFFSIZE (naive guess) and this did lead to somewhat more reasonable DT on the scope, but I didn't get any valid data b/c datascan = 0. Will keep trying tomorrow.
The original code fadc_list_save19oct.c was restored to fadc_list.c and compiled before I left. My test versions are *try*.c. Note, these versions would probably corrupt the decoder. I think the decoder may need to be modified to look for a header word to start the FADC readout. This would make it more robust against changes in the readout.
There was also a rash of annoying "not reporting" errors from CODA components. I attempted to write a well-posed question about these for the DAQ group. D. Abbott's answer is that we should use CODA 2.6.2. This we need to do anyway, in order to be able to use adaq1. I'll try this week.
Date : 23 Oct 2012 (R. Michaels)
Today I tried a bit more to read out multiple blocks (see Oct 22). No matter what, datascan is zero. This may have something to do with using older fadcLib because we have an older FADC hardware unit -- or something. It is not clear. But, this is exactly why one must push. Theoretical performance is not always matched by real life.
Next, I ported the DAQ to CODA 2.6.2 and the "adaq1" computer (without the "L"). And it works !
All components tested: EB, ER, rcgui, platform, et_start, dbedit, cedit. They all work now after some struggles (thanks partly to DAQ group and Ole Hansen).
Good news: the 58 second "not reported" messages have disappeared. D. Abbott said they probably would.
Some details: It's the TryAdaq1 config; similar to ComptonIntel. It's still on adev account, so .tchsrc was changed (old one saved). "startcoda" was modified to sense what computer; I think it would still work on adaql1 (using ComptonIntel) but I didn't try this lately. I also had to copy CODA 2.6.2 to the hallasfi1 board, and modify the setup script there. Data are written to /adaq1/data1. It is a big disk. Small glitches: I could not write to /adaqfs/coda (I think I need adaql3 to be up; it has root access), so I simply used /site/coda. Also, /adaqfs/appsroot/contrib is not "seen" by adaqfs; caused some minor trouble with kcoda.
Date : 24 Oct 2012 (R. Michaels)
I got the buffering to work (see fadc_try3.c, which I copy to fadc_list.c and compile).
Caution: We still should check the data to be sure, because it's possible the data are all garbage (I've seen this before under certain circumstances). The decoder does not yet handle these events. However, looking at the data with xcefdmp, it's clear the bits are fluctuating with time and the event length is about right for the window on the FADC, so that's a good sign.
Here is how the deadtime scaled. Note, "short read" is the readout code when we skip faReadBlock. "long read" is when we do faReadBlock
BUFFSIZE ..... Num words ...... DT of short read ..... DT of long read ...... avg DT 10 ........... 3936 .............. 5 usec ........... 110 usec ............ 16 usec 1 ............ 400 .............. N/A ............... 32 .................. 32 5 ............ 1984 ............. 5 ................. 74 .................. 19 20 ........... 7924 ............. 5 ................. 200 ................. 15 50 ........... 19596 ............ 5 ................. 464 ................. 14
This is with 2eSST and all but one channel disabled. Window size = 50. The number of words fluctuated a few percent.
For the latter case, the DT per word is 24 nsec, which is roughly consistent with the expected 40 nsec. Also, note that for 14 usec, the max rate is 71 kHz. Better, but not what we want yet.
Date : 26 Oct 2012 (R. Michaels)
Today I tried to make the BUFFSIZE even bigger, but I quickly found a limit. If BUFFSIZE=100 the DAQ was stable. 39.3K words per long read, with 910 usec DT. If I made BUFFSIZE=150 or bigger, ROC9 would segfaul, and also the EB would seqfault.
Interestingly, if ROC9 segfaults like this, you cannot do another run until you reboot the Intel PC. There is a message vmeBusLock: WARN: Previous owner of vmeBus (mutex) died unexpectedly
After rebooting the Intel PC, the DAQ works.
Not a very good day ...
