Compton DAQ Dev
From Hall A Wiki
Here are some notes on the project of developing the Compton counting mode DAQ. The original work was done mainly by Bryan Moffit and Ben Raydo in 2020, with input from others.
It is foreseen to be a precurser of a moller polarimeter DAQ as well as a counting-mode DAQ for the MOLLER project, etc.
See earlier work https://hallaweb.jlab.org/wiki/index.php/Compton_Counting_DAQ_Software_HOWTO
Contents
Budget and Plan
Budget and Plan
Upgrade of Compton and Moller polarimeter DAQ in halls A and C.
The 4 systems are approximately the same. The FADC will be modified
to support both counting mode and integrating mode.
Item Price each Qty Line cost
VXS crate 16.5 k 4 66 k
VTP 10 k 4 40 k
FADC (was not in budget)
cpu (ATOM) 3 k 4 12 k
SD 2 k 4 8 k
scaler 4 k 4 16 k
VETROC 2 k 8 16 k
TI 2 k 4 8 k
I/O 1.5 k 4 6 k
Spares 43 k
==========================================
TOTAL 215 k$
It's essentially 43k$ per DAQ (times 5 since we need spares)
except we must scrounge the FADCs. We'll buy the first system
for the hall C moller in FY2023. This 43k$ is in the FY2023 budget.
After that we may need to refine the design. Then in FY2024 we'll
purchase/build 3 more DAQs. Finally, buy spares in FY2025.
Documents
VTP manual https://userweb.jlab.org/~rom/compton/VTP-Manual.pdf
Compton Trigger https://userweb.jlab.org/~rom/compton/Compton_Trigger_2020.pdf
Current Test Setup
updated R. Michaels Oct 12, 2022
Here is a picture https://userweb.jlab.org/~rom/compton/polarimeter_daq.xfig.pdf
The basic configuration of the DAQ
VTP-Compton DAQ config: https://userweb.jlab.org/~rom/compton/haVTP_config.png
ROC1 : https://userweb.jlab.org/~rom/compton/roc1_config.png
VTP : https://userweb.jlab.org/~rom/compton/vtpROC_config.png
How to Run the Current Setup
This is a VXS crate located in the TEDF building cubicle 11. The crate is at the top of the rack. The crate was formerly installed in hall A during PREX/CREX.
Currently (Oct 12, 2022) using a 50 Hz front-panel pulser trigger on TI. Later want to modify this to use a trigger from VTP based on data from FADC and/or the VETROC.
updated Jun 8, 2023. Bryan Moffit
How to login to the two ROCs and start the ROC
ssh -X amoll1 -l sbs-onl ./startroc1
ssh -X havtp1 -l sbs-onl ./startRoc
How to start CODA on atedf3. Username adaq.
kcoda startcoda
Readout Lists:
- VTP
havtp1:~/comptonDaq/vtp/rol/vtp_list.so
- ROC1
amoll1:~/comptonDaq/rol/vtpCompton_list.so
Module configuration files:
- VTP - Compton firmware
havtp1:~/comptonDaq/havtp1.cnf
- VETROC
amoll1:~/vetroc/amoll1.cnf
- FADC250
amoll1:~/fadc250/amoll1.cnf
VTP library update
Jun 8, 2023 - Bryan Moffit
The VTP library was updated to the current version in github. Older version were moved to
~/comtponDaq/oldvtp/
Updated the LINUXVME environment variables for havtp1 in
~/.bashrc
to
export LINUXVME=${HOME}/comptonDaq/vtp
export LINUXVME_INC=$LINUXVME
export LINUXVME_LIB=$LINUXVME
export LINUXVME_BIN=$LINUXVME
Built and installed vtp library
cd ~/comptonDaq
git clone -b devel-NPS https://github.com/JeffersonLab/vtp.git
cd vtp
git rev-parse --short HEAD
a208f54
make install
Built and installed vtpFirmwareLoad program (used at boot)
cd ~/comptonDaq/vtp/firmware
make install
Built and installed DiagGui server
cd ~/comptonDaq/vtp/DiagGui
make install
Confirmed vtpFirmwareLoad configuration file:
/usr/local/vtp/params/firmwares/vtp_firmware.txt
havtp1 fe_vtp_hallb_z7.bin fe_vtp_halla_v7_compton.bin
Rebooted.
Updated the VTP payload enables for modules missing from slot 15 and 16 (in ~/comptonDaq/havtp1.cnf):
# slot: 10 13 9 14 8 15 7 16 6 17 5 18 4 19 3 20
# payload: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
VTP_PAYLOAD_EN 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0
# Before update:
VTP_PAYLOAD_EN 0 1 0 1 0 1 0 1 0 0 0 0 0 0 1 0
Ran the DAQ using Bob's instructions and was able to readout all modules, including VTP. The trigger rate appears to be 100Hz.
Updated the VETROC configuration file (needs to use full hostname):
amoll1:~/vetroc/amoll1.jlab.org.cnf -> amoll1.cnf
Archive of Problems
Oct 12, 2022.
If I drop the VTP from the readout and remove tiRocEnable(2) from amoll1:/home/sbs-onl/comptonDaq/rol/vtpCompton_list.c then the DAQ runs fine and both the FADC and VETROC data make sense. Next I added the VTP. Still a FP trigger. It downloads and prestarts. But here is where the problems start.
Dec 3, 2022
For several weeks I had bus errors. Chasing this down involved looking a lot at the library (vtpLib), the readout list (CRL), the test codes, how the firmware gets loaded, etc. In a way, it was a good thing it happened because I learned a lot. In the end, it was a simple problem. I was using a version of vtpLib which was too modern for the firmware, so the library tried to access registers that were not yet defined by that firmware. The solution is get the correct branch from github and pick a consistent set of firmware, library, test code, and readout list. These all have to "hang together" and be consistent. Well, duh ! That was the main lesson. And here are some smaller lessons. One can find the address of the bus errors by logging into the serial interface (uses minicom in our test setup). The address is printed in square brackets [0x43C02000], see vtpLib.h for what the address means, this example is pointing to DMA transfer and it means generally something is mismatched between the library vtpLib and the firmware, and specifically in this case you are running the DMA test code in ./test directory but the DMA capability is not built into that version of the firmware. A "bus error" occurs when the program is compiled with a different (wrong) versions of the VTP library. There are two firmwares for 2 chips on the VTP. One is Z7 and the other is V7. Z7 handles the CODA roc. The coda_roc it can be a hardware roc (used by GEMs for high-speed readout) or a software roc (was used by Compton). A 3rd development for Z7 is streaming readout (not used in hall A yet). The other chip (V7) is used for Compton specific application and registers. These firwmare are specified in havtp1:/usr/local/vtp/params/firmwares/vtp_firmware.txt It looks like our current problem is that we probably picked up the newest (or relatively new) library but this is not compatible with the firmware for software roc on Z7 and maybe not for Compton on V7 chip. There are various vtpLib available. 1. Compton version 2. hall B compton version 3. devel-NPS -- the latest as of today 4. probably others (?) A goal is to have one library work for all firmware. Two possible solutions. 1. Obtain the version of the Compton vtpLib library (instead of the newest lib) This is what I did, and it worked. 2. (longer term) Try a "devel-NPS" version and compatible firmware -- Ben and Bryan will pursue this. Even though solution 2 is preferred, solution 1 was a valuable learning point.
See Current Problems instead. The problem ended up being simple. For the time being I leave this here, although it's pretty much noise.
Specific problem with vtpLib. This affects both the Compton
version and the HCAL version. I think if I fix this I will
understand the rash of "Bus error"s and "Segmentation Fault"s
when running CODA.
This code causes a Bus Error. Actually lots of code causes Bus Error
but let's focus on this one. BTW, the Chip 0 FW type is 15
int
vtpGetFW_Type(int chip)
{
int rval=0;
CHECKINIT;
if(chip == 1) {
VLOCK;
printf("here 1111\n");
rval = vtp->clk.FW_Type;
// NOTE !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// The following line never happens because of a Bus Error.
printf("here 2222 chip %d rval %d\n",chip,rval);
VUNLOCK;
}else{
VLOCK;
rval = vtp->v7.clk.FW_Type;
VUNLOCK;
}
return rval;
}
Running vtpStatus causes a Bus Error when asking Chip 1 for the vtp->clk.FW_Type
[sbs-onl@havtp1 test]$ ./vtpStatus size = 131072 vtpSPIOpen: spi mode: 0x0 bits per word: 8 max speed: 781250 Hz (781 KHz) vtpCheckAddresses: ---------- Checking VTP register map ---------- vtpLock: ERROR: EOWNERDEAD vtpLock: WARN: Previous owner of VTP (mutex) died unexpectedly Attempting to recover.. Successful! vtpInit: Setting up VTP PLL for 250MHz VXS reference si5341_checkBasePart: ID=5341 si5341_readStatus: status = 00, OK SYSINCAL 0, OK LOSXAXB 0, OK LOSREF 0, OK LOL 0, OK SMBUS_TIMEOUT 0, OK vtpV7PllReset: PLL successfully locked vtpV7PllReset: PLL successfully locked vtpV7PllLocked: PLL successfully locked here 1111 Bus error
Comment re: status of vtpLib
There is not much difference between the Compton version and the HCAL version. Mainly printf statements and serdes checks. In both the Compton version and the HCAL version, the vtpLib was missing vtpEbBuildTestEvent(int len) and vtpEbReset(). Such as it was, the ./test codes would not build (undefined reference). I found these subroutines elsewhere and put them in by hand.
Nov 9, 2022 (Nov 10 below) Current status, globally, is that without the VTP the DAQ runs with a front panel trigger at 50 Hz and reads the FADC and VETROC. Data looks ok. Adding the VTP causes problems.
See READOUT LIST .... https://userweb.jlab.org/~rom/compton/vtp_list_9Nov22.c
Problems today (Nov 9, 2022) 1. There are complaints that I cannot vtpSetCompton_EnableScalerReadout using firwmare 15. Probably the least of my problems. 2. Something wrong with VTP Serdes links. It takes several seconds to time out. Waiting on links: PP1 PP2 PP3 PP4 ... (etc) vtpSerdesCheckLinks: ERROR - all serdes links not up! 3. What library to use ? vtpEbResetFifo defined in /mnt/SBS/coda/3.10_arm/Linux-armv7l/lib/libvtp.so but not in /home/sbs-onl/comptonDaq/vtp/libvtp.so I'm told not to use /mnt/SBS/coda/3.10_arm/Linux-armv7l/lib/ Pulling the libvtp from the SBS HCAL setup, I see that it is also quite different. Which one to use ? Generally, the fact that the readout code I got referred to routines that are not in my libraries, I assume I must have a mismatch of readout code and library from the Compton polarimeter. 4. Bus errors for (I think) any code with the string "Ti" in it. See "Bob1" in the readout list to show the lines that are commented out. It looks like any code with the string "Ti" in it (presumably tries to talk to the TI) causes a bus error. 5. To USE_DMA or not ? Different behavior whether I do #define USE_DMA or //#define USE_DMA Whether I USE_DMA or not ... With all the lines commented out that causes bus error, the DAQ goes into an active state but usually gets no events i.e. rocTrigger() is not entered. The DAQ does go through all the transitions without problems. if those lines are commented out. Again, see "Bob1" in the readout list. Update Nov 10, 2022 I got the test code in havtp1:/home/sbs-onl/comptonDaq/vtp/test to compile using, I think, the recommended vtpLib and header, and no mixing of versions. Aren't the codes supposed to just work ? Unfortunately, depending on the code, the either do a "Bus error", a "segmentation fault" or report an error like vtpLock: ERROR: EOWNERDEAD vtpV7SetReset: ERROR: VTP not initialized Chip #0 reports the firmware type is 15 (what we want, right ?) Chip 0 FW type 15 But when I try to address Chip #1 it does a Bus error, specifically on this line rval = vtp->clk.FW_Type; and never makes it to the next line. I like the idea of these test codes. You can learn a lot, like how the configuration works (that seems to work). I know everyone is busy with more important tasks, but it does seem fair to ask for a hardware VTP to go along with a working set of test codes. However, maybe I screwed them up somehow. ---------------------------------------------------------------------------------
Analysis Software
Nothing here yet. Work in progress. But there was software for the DAQ used during PREX/CREX.
Old Results
- Background information: "https://userweb.jlab.org/~rom/Compton_info"
- Results from PREX/CREX "https://userweb.jlab.org/~rom/Compton_ML" (me bored during pandemic)
- Older Material "https://userweb.jlab.org/~rom/compton_old"
