HV HowTo for Experts
High Voltage in Hall A
First of all, please be aware of the simple instructions for users: HowTo for Users
For the SBS experiments, the high voltage is controllable with EPICS: See SBS EPICS
Contents
Overview of Architecture
The HV crates sit in the hall, e.g. on the detector stacks. A set of cards with (usually) 12 channels of negative or positive HV are inserted into the crates. A custom "serial board" (built by Javier Gomez and Jack Segal) talks to the cards. This "serial board" replaces an old, obsolete motherboard. (There might be a few of these old crates somewhere, but I doubt it.) A Perl "Shim" server (written by Brad Sawatzky) runs on a PC (usually a Raspberry Pi) nearby the HV crate. The "Shim" server uses (via a socket connection) a low-level C code written by Javier to talk to his serial card in the HV crate. On the User end, a Java GUI (written by Roman Pomatsalyuk) displays the HV information and provides User control. This Java GUI talks to the Shim server. Alternatively, the Java GUI can talk to the motherboard via a Portserver. A relatively new development (2021) is an EPICS layer. The Shim server can talk to the Java GUI as well as the EPICS layer. The EPICS support (based on the old LeCroy HV EPICS support developed by Chris Slominski and modernized by Stephen Wood) has been added to the rPI boards in the crates used for SBS. The EPICS server on each rPI communicates with the shim which has been upgraded to support simultaneous connections from EPICS and the Java GUI. The addition of EPICS support allows the use of the broad range of EPICS tools for GUIs, stripcharts and archiving. See SBS EPICS for more information
Existing Crates
Here is a list of HV crates in Hall A as of May 2021.
The crates that use portservers are talking to the old motherboard. The crates that use an rPI PC with Shim have had their motherboards removed. See Architecture above.
Location rPi or Portserver for Shim Config on ~/slowc How to start on aslow Left HRS (1 crate) rpi8 LEFT cd ~aslow/slowc ; ./hvs LEFT BigBite detectors rpi17 Ecal, Grinch rpi18 SBS Hcal rpi20 rpi21 Beamline portserver hatsv4: 2003 BEAMLINE cd ~aslow/slowc ; ./hvs BEAMLINE (Not working 5/26/2021)
This following list, from September 2014, is obsolete:
Location Portserver, or PC for Shim Config on ~/slowc How to start on aslow Left HRS (1 crate) rpi8 LEFT cd ~aslow/slowc ; ./hvs LEFT Right HRS (bottom crate) rpi7 RIGHT Right HRS (top crate) rpi4 RIGHT cd ~aslow/slowc ; ./hvs RIGHT (this starts both R-HRS crates) Beamline portserver hatsv4: 2003 BEAMLINE cd ~aslow/slowc ; ./hvs BEAMLINE
Restarting the Servers that include EPICS (2021)
This section applies to those crates that have been updated to include the EPICS server.
Servers run on each Raspberry PI PC's connected to a Lecroy HV mainframes. They are a low level server that communicates with HV cards in the mainframe, a "shim" that communicates with the low level server and presents an interface that mimics the original mainframe interface, and and EPICS server that allows control and monitoring by EPICS applications.
A cron script ensures that the servers are running on on pi account on each rPI.
$ crontab -l 2,6,10,14,18,22,26,30,34,38,42,46,50,54,58 * * * * /home/pi/scripts/start_hv_cron
The start_hv_cron script checks that the servers are running. If they are running, the script does nothing. If any of the servers are not running, the script starts them. This means that the servers should always be running (within about 5 min of restarting the rPI.) and you should not have to do anything.
The start_hv_cron also configures the EPICS setup so that EPICS PVs (process variables) are defined for each HV channel in each slot that is installed in the crate.
If the servers need to be restarted, that can be done in one of three ways.
1. Power cycle the mainframe. 2. Reboot the rPI by logging on to the pi account and typing "sudo reboot" 3. Killing the servers by logging on the pi account and typing "~/scripts/kill_hv_servers"
In each of these cases, the servers will be restarted by cron within four to five minutes.
Restarting the Servers
The servers can run on a PC with a serial connection. As of Sept 2014, all the HRS servers are running on Raspberry PI PCs.
A cron script ensures that the server is running on pi@rpi
$ crontab -l
# cron jobs 5,10,15,20,25,30,35,40,45,50,53,55,58 * * * * /home/pi/scripts/start_hv_cron
The start_hv_cron script checks if the server is running. If it is running, the script does nothing. If it is not running, the script starts it. This means that the server should always be running (within about 5 min of restarting the rPI.) and you should not have to do anything.
A simple way to restart the server is to reboot the rpi. The cron script will restart it.
If you prefer to be faster here's how
ssh into the rpi of interest (rpi4 or rpi7 on R-HRS or rpi8 on L-HRS). User is "pi" and sorry I can't say the password here. I'll put it near the whiteboard.
These should be typed at the command prompt
ps -awx | grep -i shim kill -9 27913 (assuming the PID of the process was 27913) then /home/pi/scripts/start_hv_cron exit
Next, on adaqsc, or wherever you run the Java GUI, and using the aslow account: kill the Java Gui and restart it
cd /adaqfs/home/aslow/slowc ./hvs LEFT or ./hvs RIGHT
May 2022 updates the Raspberry Pi
Some goals of the recent work on the software for HV controls were: 1) update the software so that HV can work on newer rpi3, 4. 2) make several backup SD cards so that we can rapidly swap them to fix broken SD cards (presumed to be zapped by radiation).
Generally the most recent version is in a directory like /soft1 (could be /soft2, /soft3, etc)
within /soft1 are some directories
/scripts
contains the script(s) that launch the whole thing. Mainly you need start_hv_cron, as explained above. configure_epics.py configures the EPICS. Currently there is a hack to force the crate # to be 8 instead of deriving it from rpi22.jlab.org. See "hack" in that script.
/lecroy_servers
Contains the C code that talks to the crate and the Perl "shim" server. -- pay attention to GPIO memory offset, it depends on version of rpi
20220504_i2lchv_rPI-linux driver to talk to HV crate 20220504_i2lchv_rPI-linux.c source of driver Note about rpi3: need to use /dev/serial0 instead of /dev/ttyAMA0. Also, don't forget to enable the serial lines (raspi-config --> Interface options --> Serial interface --> enable, also explained below) LecroyHV_shim-01May2022-epics Perl "shim" server
/epics
The EPICS layer
/pkg_to_install
See the README. But it was probably already installed.
In /home/pi there are links to these, e.g. lecroy_servers -> ./soft1/lecroy_servers/
Setting Up A New Installation
For the rPI, the installation kit is in /mss/home/rom/rpi*tar*.
There is typically a READMEpi or README in each directory. See also the notes above (updates to rpi).
I will update this on github. There are two repositories.
1. Frontend C code and Perl Shim server (credit: Javier Gomez wrote the C code and Brad Sawatzky wrote the Perl server.)
2. Java GUI (credit: Roman Pomatsalyuk wrote the Java GUI.)
3. Not sure about EPICS. I have to learn this part (noted, May 2022).
You should always pick the latest date tar file, or use Git. For the tar file the date is usually written in the filename.
Detailed steps 1. Format SD card using formatter software for rpi. 2. Download the rpi image (operating system) on SD card using a saved image. 3. Enable rpi on network -- ask CST, need to give mac address and add to DHCP. 4. password for "pi" account set to our hall A/C value. 6. Enable sshd on raspberry (Google how). In the same config menu, you can enable serial port. (step 12) 6. scp rom@jlabl2:/home/rom/rpi_setup_6May22.tar . or whatever is the latest version. 7. tar xvf this on rpi 8. Descend into each /pkg_to_install subdir and do this: sudo perl Makefile.PL ; sudo make ; sudo make test ; sudo make install 9. Might have to build and install some items like readline and ncurses "by hand" from pkg_to_install cd readline-6.3/ sudo make libreadline.a cd ncurses-5.9/lib sudo make libncurses.a cd /usr/lib sudo cp /home/pi/pkg_to_install/readline-6.3/libreadline.a . sudo cp /home/pi/pkg_to_install/ncurses-5.9/lib/libncurses.a . cd /usr/include sudo ln -s /home/pi/pkg_to_install/readline-6.3 readline sudo ln -s /home/pi/pkg_to_install/ncurses-5.9 ncurses 10. For EPICS, in /home/pi/scripts need to run configure_epics.py. I think this is automatic. You get EPICS variables like HAHV8:Pw where the "8" should be replaced by the IP number, e.g. rpi22 produces HAHV22:Pw. To check: From rpi22, you can do “telnet localhost 20000” and type the command “dbl”. That should list all the EPICS variables 11. Create crontab entry as per /home/pi/scripts/cron.tab 12. (edit: June 2024). It seems necessary to enable the serial connection. Login to the rpi and then do this: sudo raspi-config 3. Interface Options I6 Serial interface First question (login shell ?) No Second question (enable serial port hardware ?) Yes After the above, you should reboot. Then /dev/serial0 will appear. 13. It seems to be a good idea to sync everything and stop the errors when compiling, by running the following command once when connected to the network: sudo date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z" Done ! (gulp)
What can go wrong ?
Common troublshooting items are listed in the User Guide (search for troubleshooting)
https://hallaweb.jlab.org/wiki/index.php?title=HV_HowTo_for_Users
Below is a list of other problems I've seen and their solution.
1. Power-cycling a HV crate has been known to help. Especially if more than one HV card does not respond. Then it's probably a crate problem not a card problem.
2. No connection to server. Try restarting the server. Instructions in "Restarting the Server" above.
3. It happened to me a couple times that, after power-cycling etc, I could not connect to the LeCroy cards. When I ran the code in /home/pi/hvctrl it somehow "woke up" the crate and then the software listed here worked. I know this sounds like voodoo. Anyway, see the README in /home/pi/hvctrl. It is an alternative, very primitive and simple control system.