|
|
| Line 1: |
Line 1: |
| − | __TOC__
| + | This page has moved to the [https://sbs.jlab.org/wiki/index.php/SBS_EPICS SBS Wiki]. |
| − | | + | |
| − | New EPICS controls for high voltage control/monitoring and for GEM gas system monitoring have been developed for the SBS experiments.
| + | |
| − | | + | |
| − | = High Voltage =
| + | |
| − | | + | |
| − | The High voltage for PMTs and GEMs for SBS experiments use at least 4
| + | |
| − | different types of power supplies. These include:
| + | |
| − | | + | |
| − | * 5 Lecroy mainframes
| + | |
| − | * A Caen 1527 mainframe
| + | |
| − | * A Wiener Mpod mainframe for (UVA) GEM detectors
| + | |
| − | * CAEN VME V6521 cards for (INFN) GEM detectors
| + | |
| − | * CAEN 4527 mainframe(s) (possible use for GeN-RP)
| + | |
| − | | + | |
| − | Each of these systems has a vendor or lab developed graphical or web
| + | |
| − | interface. In order to simplify operations, EPICS controls have been
| + | |
| − | implemented for all these systems.
| + | |
| − | | + | |
| − | == User control of high voltage ==
| + | |
| − | | + | |
| − | EPICS Graphical interfaces have been implemented for all the HV under EPICS
| + | |
| − | control. These interfaces include the ability to [[#Backup and Restore|backup and restore]]
| + | |
| − | HV settings.
| + | |
| − | | + | |
| − | The various types of high voltage mainframes have vendor or user
| + | |
| − | supplied control interfaces that don't use EPICS. These interfaces
| + | |
| − | should continue to work.
| + | |
| − | | + | |
| − | To start the HV GUI controls, logon to "aslow@adaqsc" and type the command
| + | |
| − | | + | |
| − | go_hv
| + | |
| − | | + | |
| − | This will bring up a main screen from which a desired detector or sub-detector can be selected. Clicking on one of the detector buttons will give a choice of "Listed Controls & Monitor" or "Histogram Monitor." Choosing the listed controls option will bring up a table of actual voltage and currents, and fields in which the voltage, trip current, maximum voltage and ramp rates can be changed. | + | |
| − | | + | |
| − | [[File:HV main screen.png|300px]] [[File:HV control screen.png|600px]]
| + | |
| − | | + | |
| − | For a graphical view of the high voltage, choose the second option when clicking on a detector button. This will show a bar chart of voltage and current for all the channels.
| + | |
| − | | + | |
| − | [[File:HV monitor.png|600px]]
| + | |
| − | | + | |
| − | == Backup and Restore ==
| + | |
| − | High voltage settings may be backed up and restored from the HV GUI. Select "Backup" from the "Backup /Restore" button. HV backups, which are readable text files, will be placed in <code>~aslow/EPICS/HV/backup-dir</code>. "Restore" should be used with care. It is recommended to only restore HV settings for a single detector or portion of a detector at a time. This may be done by restoring a copy of a backup file where all the lines except for the detector of interested have been removed.
| + | |
| − | | + | |
| − | == HV Alarms ==
| + | |
| − | For each HV channel, a voltage difference (VDiff) EPICS signal is provided. Alarm limits are places on
| + | |
| − | these signals so that shift workers will be notified via the alarm handler if a channel trips off or
| + | |
| − | if the actual voltage provided by the channel differs from the set voltage. By default, a minor alarm is raised if the monitored voltage is more than 25 volts above or below the set voltage, and a major alarm is raised
| + | |
| − | if the voltage is more than 50 volts away from the set voltage. These limits may be changed for individual channels
| + | |
| − | using the alarm handler.
| + | |
| − | | + | |
| − | == Building GUI screens ==
| + | |
| − | | + | |
| − | If detector HV is changed due to bad channels or other reasons, control
| + | |
| − | screens will need to be rebuilt. The configuration files are the HV.hvc and HV.group files located in subdirectories of <code>~aslow/EPICS/HV</code>. The relevant subdirectories are presently bb_gem, bb_hodo, bb_other, lhrs_fp, and sbs_hcal. The subdirectories are defined by the Makefiles in the "all" subdirectories, bb_all, sbs_all and lhrs_all.
| + | |
| − | | + | |
| − | The HV.hvc in each subdirectories defines the HV for one or more detectors. Each line of the file is of the form:
| + | |
| − | | + | |
| − | ElementName Crate# Slot# Channel# Group# Row Column
| + | |
| − | | + | |
| − | The group numbers are defined in the HV.group files. The set of HV channels in each group are accessed by corresponding button on the HV GUI. The Row and Column can indicate vertical and horizontal position of the element within a detector such as a calorimeter or they may be set to "-1 -1".
| + | |
| − | | + | |
| − | If an HV.hvc file is edited, type make within the directory containing the modified file. Then type make from <code>~aslow/EPICS/HV</code>. This will rebuild the HV GUI.
| + | |
| − | | + | |
| − | After rebuilding the GUI, all running instances of the GUI should be restarted.
| + | |
| − | | + | |
| − | == Crate Numbering ==
| + | |
| − | | + | |
| − | Every high voltage mainframe must be assigned a unique crate/mainframe
| + | |
| − | number. For the Lecroy crates, this crate number is automatically
| + | |
| − | assigned from the Raspberry Pi hostname. If the hostname is of the
| + | |
| − | form rpiNN, then NN is the crate number. Other HV system types are
| + | |
| − | assigned a crate number manually. The following mainframes are in use
| + | |
| − | for the SBS experiments:
| + | |
| − | | + | |
| − | 0 CAEN1527 BigBite Hodoscope
| + | |
| − | 8 Lecroy HRS Left
| + | |
| − | 17 Lecroy BigBite
| + | |
| − | 18 Lecroy BigBite
| + | |
| − | 20 Lecroy Hcal
| + | |
| − | 21 Lecroy Hcal
| + | |
| − | 91 MPOD UVA GEM HV
| + | |
| − | 92 CAEN V6521 INFN GEM HV
| + | |
| − | 93 CAEN4527 GeN-RP
| + | |
| − | | + | |
| − | == EPICS High Voltage PV Naming Convention ==
| + | |
| − | | + | |
| − | The EPICS Process Variables (PVs) for individual HV channels are
| + | |
| − | chosen to have a consistent form, independent of hardware. The form is
| + | |
| − | chosen to match the built in EPICS on CAEN4527 mainframes. The use of
| + | |
| − | consistent PV naming convention allows for GUIs that are independent
| + | |
| − | of hardware.
| + | |
| − | | + | |
| − | EPICS PVs for HV have the form
| + | |
| − | | + | |
| − | "%s%d:%.2d:%.3d:%s", prefix, crate, slot, channel, property
| + | |
| − | | + | |
| − | where prefix is a string unique to Hall A ("HAHV"), crate is a
| + | |
| − | mainframe number (from 0 to 99), slot is a card # within a mainframe
| + | |
| − | (0-99), channel is the channel number within a card (0-999) and
| + | |
| − | property is the attribute to be written or read. For example, the monitored voltage (VMon property)
| + | |
| − | for crate 7, slot 5, channel 3 will have the PV "HAHV7:05:003:VMon".
| + | |
| − | | + | |
| − | Properties common to all hardware types are:
| + | |
| − | | + | |
| − | V0Set (w) Voltage setting for a channel
| + | |
| − | V0Setr (r) Readback of voltage setting
| + | |
| − | VMon (r) Monitored/measured voltage
| + | |
| − | I0Set (w) Setting for Current at which to trip channel
| + | |
| − | I0Setr (r) Readback of trip current setting
| + | |
| − | IMon (r) Measured current
| + | |
| − | Pw (w) (0/1) Turn off or on the channel
| + | |
| − | Pwr (r) Readback of on/off status
| + | |
| − | RUp (w) Ramp rate in V/s when turning on channel
| + | |
| − | Rupr (r) Readback of ramp up rate
| + | |
| − | RDWn (w) Ramp down rate in V/s when turning off channel
| + | |
| − | RDWnr (r) Readback of ramp up rate
| + | |
| − | VDiff (r) Difference between set and monitor voltage. Used for alarms.
| + | |
| − | | + | |
| − | These property names are chosen to be consistent with the property
| + | |
| − | ppnames of the CAEN4527 built in EPICS.
| + | |
| − | | + | |
| − | = GEM Gas Controls =
| + | |
| − | | + | |
| − | The Detector Support Group (DSG) has developed a system using small sensors
| + | |
| − | to detect the gas flow of each channel of the GEM gas distribution system | + | |
| − | (also developed/designed/assembled by DSG). The SBS system is based on the
| + | |
| − | BigBite system which is running in Hall A right now. The BigBite system is
| + | |
| − | an 8 channel system, while the the SBS will have 42 channels.
| + | |
| − |
| + | |
| − | The system was designed to provide flow values for each individual GEM
| + | |
| − | layer as opposed to using a single mass flow meter, which would become
| + | |
| − | cost prohibitive to provide 50 channels of flow.
| + | |
| − | | + | |
| − | These systems can be monitored at [https://epicsweb.jlab.org/wedm/screen?edl=%2fcs%2fopshome%2fedm%2fdsg%2fSBS_GEM_flow_press.edl SBS GEM Flow Readout] and [https://epicsweb.jlab.org/wedm/screen?edl=%2fcs%2fopshome%2fedm%2fdsg%2fGEM_text.edl BigBite GEM Gas Flow Readout] screens.
| + | |
| − | | + | |
| − | For system details see these DSG publications:
| + | |
| − |
| + | |
| − | [https://www.jlab.org/sites/default/files/dsg/techDocs/Talks/Gas%20Distribution%20and%20Monitoring%20System%20for%20the%20GEM%20detectors_final.pdf DSG development of the GEM gas distribution and readout system]
| + | |
| − | | + | |
| − | [https://www.jlab.org/sites/default/files/dsg/techDocs/Talks/gem%20prototype%20progress_020821_submit.pdf DSG prototype gas distribution system with remote monitoring]
| + | |
| − | | + | |
| − | [https://www.jlab.org/sites/default/files/dsg/notes/2021-12_post.pdf DSG Gas flow monitoring for Hall A GEM]
| + | |
| − | | + | |
| − | [https://www.jlab.org/div_dept/physics_division/dsg/notes/2019-33%20Design%20of%20the%20Gas%20Distribution%20System%20for%20the%20Hall%20A%20GEM%20Detectors.pdf DSG GEM gas distribution system]
| + | |
| − | | + | |
| − | [https://www.jlab.org/div_dept/physics_division/dsg/notes/2020-20%20Development%20of%20Readout%20Electronics%20for%20Gas%20Flow%20Sensors%20for%20the%20Hall%20A%20GEM%20Detectors.pdf DSG GEM readout electronics]
| + | |
| − | | + | |
| − | [https://www.jlab.org/div_dept/physics_division/dsg/notes/2020-32.pdf DSG designed flow sensor chassis]
| + | |
| − | | + | |
| − | The GEM detector Gas distribution system pushes information to EPICS
| + | |
| − | for monitoring. The EPICS signals are on the IOC iocGEMGas running
| + | |
| − | the GEMGasApp on adaqsc.
| + | |
| − | | + | |
| − | = EPICS Server Software =
| + | |
| − | | + | |
| − | == EPICS Software ==
| + | |
| − | | + | |
| − | The master copy of the EPICS HV controls for SBS are in the directory
| + | |
| − | <code>~aslow/EPICS</code> on adaqsc. The <code>EPICS</code> directory contains the following
| + | |
| − | directories:
| + | |
| − | | + | |
| − | ; base-3.15.8 : The base EPICS distribution from https://epics.anl.gov/
| + | |
| − | ; clas12-epics-third-party-libs : Contains CAEN libraries that are needed for the communicating with SY1527 HV crates
| + | |
| − | ; sbs-epics : Contains the source code for EPICS support to provide IOCs (Input Output Controllers) for all the models of HV crates in use. Also contains configuration files to define EPICS signals for all the HV channels in use.
| + | |
| − | | + | |
| − | The base-3.15.8 and sbs-epics directories are also installed on the
| + | |
| − | Raspberry Pi for each Lecroy Mainframe and on intelbbmpd for controlling
| + | |
| − | CAEN VME HV cards.
| + | |
| − | | + | |
| − | <code>sbs-epics/apps</code> contains these source code/configuration directories:
| + | |
| − | | + | |
| − | ; Alias4527 : Mirrors 4527 crate signals to provide "fake" readback signals. (See [[#CAEN 4527|CAEN 4527]] section below)
| + | |
| − | ; caenHvApp : Code and configuration for CAEN SY1527 mainframes. (Used by BigBite hodoscope)
| + | |
| − | ; caenV65XXApp : Code and configuration for CAEN VME HV cards. (Used by GEM detectors
| + | |
| − | ; iocBoot : Scripts to start the IOCs for each of the HV systems
| + | |
| − | ; lecroyApp : Code and configuration for RPi enabled Lecroy Mainframtes
| + | |
| − | ; procServ:2.6-0App : A wrapper for running the EPICS IOCs
| + | |
| − | ; snmpApp : Code and configuration for communicating with SNMP systems. Used for Wiener HV.
| + | |
| − | ; syX527App : Library routines needed by caenHvApp
| + | |
| − | ; GEMGasApp : Soft Channel records for BB Gas system
| + | |
| − | | + | |
| − | This code is compiled by running make from sbs-epics/apps. The
| + | |
| − | Makefiles in some of the above source code directories are setup to
| + | |
| − | compile only under certain conditions:
| + | |
| − | | + | |
| − | caenHvApp and syX527App will only be compiled if the
| + | |
| − | <code>clas12-epics-third-party-libs</code> directory exists.
| + | |
| − | | + | |
| − | snmpApp will only be compiled if snmp support is installed on the
| + | |
| − | operating system. (By checking if
| + | |
| − | <code>/usr/include/net-snmp/net-snmp-config.h</code> exists.)
| + | |
| − | | + | |
| − | caenV65XXApp is only compiled if '''LINUXVME_INC''' is defined.
| + | |
| − | | + | |
| − | == ProcServ and ProcServMgr ==
| + | |
| − | On systems where the EPICS IOCS (servers) run, a cron task is
| + | |
| − | configured to periodically run ProcServMgr. ProcServMgr looks at the
| + | |
| − | file <code>sbs-epics/apps/iocBoot/procServMgr.conf</code> to determine which IOC
| + | |
| − | tasks need to run on that system and starts any IOCs that are not
| + | |
| − | running. The IOC tasks are run by ProcServ which allows experts to
| + | |
| − | login to the IOC with the command <code>telnet PORT</code> where the
| + | |
| − | PORT number is defined in <code>ProcServMgr.conf</code>.
| + | |
| − | | + | |
| − | == Hardware specific notes ==
| + | |
| − | | + | |
| − | ===Lecroy===
| + | |
| − | | + | |
| − | Hall A has a large number of Lecroy Mainframes. These were controlled
| + | |
| − | either by ARCnet or RS232. Several years ago, the PC boards in each
| + | |
| − | of these mainframes was replaced with Raspberry Pi boards that
| + | |
| − | communcate with the mainframe via a custom serial board developed by
| + | |
| − | Javier Gomez and Jack Segal. Two processes run on each board. A process
| + | |
| − | that communicates with the HV cards (written by J. Gomez) and a
| + | |
| − | process (shim) (written by Brad Sawatzky) that mimics the original
| + | |
| − | ARCnet/RS232 command set, except that it uses TCP/IP. This is further
| + | |
| − | described in [[HV HowTo for Experts]].
| + | |
| − | | + | |
| − | In 2000, Chris Slominski developed an EPICS driver for the Lecroy
| + | |
| − | mainframes that communicated with ARCnet. This driver has been
| + | |
| − | adapted to use the TCP/IP interface that the modified Lecroy
| + | |
| − | mainframes now provide.
| + | |
| − | | + | |
| − | A single linux IOC can be configured to support multiple mainframes or
| + | |
| − | an IOC can be run on each rPI board. For the SBS HV, the later method
| + | |
| − | is used. Each mainframe is configured to create create EPICS signals
| + | |
| − | for the installed HV cards.
| + | |
| − | | + | |
| − | To install EPICS on a Lecroy Mainframe, use the following steps.
| + | |
| − | | + | |
| − | # Setup a rpi with a hostname of rpiN, where N is not an<!--
| + | |
| − | --> existing HV crate number (See "Crate Numbering" section).<!--
| + | |
| − | --> (Unless it is replacing an existing crate.)
| + | |
| − | # Insure that the rPI board in the mainframe has the<!--
| + | |
| − | --> software prerequisites to compile C programs (gcc,<!--
| + | |
| − | --> ncurses, readline) and the perl modules needed by the<!--
| + | |
| − | --> shim (IO-Multiplex, Net-Server, IO-Tty, Expect). If<!--
| + | |
| − | --> the JAVA GUI already works with this mainframe,<!--
| + | |
| − | --> then these prerequisites are installed. If they<!--
| + | |
| − | --> are not, ask the Spectrometer Support Group to<!--
| + | |
| − | --> prepare an RPI with these prerequisites.
| + | |
| − | # Obtain the files scripts.tar.gz, lecroy_servers.tar.gz<!--
| + | |
| − | --> and epics.tar.gz and place them in the home<!--
| + | |
| − | --> directory of the pi account on the rPI. These<!--
| + | |
| − | --> can be found in the directory <code>~/EPICS/RPI</code> of the<!--
| + | |
| − | --> <code>aslow@adaqsc</code> account.
| + | |
| − | # Untar all three files into the pi account home directory.
| + | |
| − | # Type <code>crontab -r</code> and <code>~/scripts/kill_hv_all</code> to kill<!--
| + | |
| − | --> and disable old versions of the Lecroy software.
| + | |
| − | # Type <code>crontab ~/scripts/cron.tab</code>
| + | |
| − | # Wait 5 minutes and check that an "i2lchv" process<!--
| + | |
| − | --> and a "LecroyHV_shim" procees is running.
| + | |
| − | # Check that the EPICS server is running with<!--
| + | |
| − | --> <code>telnet localhoset 20000</code>. You should see the<!--
| + | |
| − | --> <code>epics></code> prompt. Typing <code>dbl</code> should show a long<!--
| + | |
| − | --> list of EPICS signal names starting with "HAHV91".
| + | |
| − |
| + | |
| − | ====Lecroy Cron job====
| + | |
| − | | + | |
| − | The cronjob that is setup with <code>crontab ~/scripts/cron.tab</code> run the
| + | |
| − | script <code>start_hv_cron_new</code> every four minutes. This script checks
| + | |
| − | that each of the servers, i2lchv, the shim and the epics server is
| + | |
| − | running. It will start any server that is not running.
| + | |
| − | | + | |
| − | If the EPICS server is not running, <code>start_hv_cron_new</code> will first run
| + | |
| − | <code>~/scripts/configure_epics.py</code>. This python script does several
| + | |
| − | things.
| + | |
| − | | + | |
| − | # Determines the mainframe/crate number from the hostname rpiN.
| + | |
| − | # Reads the file <code>/home/pi/scripts/LecroySlotList.txt</code> to find which # slots are occupied and what type of module is in each slot. Currenly, only 1461N, 1461P and 1471N cards are supported.
| + | |
| − | # Using the slot information, define epics signals for every HV # channel in the crate by writing the file <code>LecroyHV.substitutions</code> to <code>/home/pi/epics/sbs-epics/apps/db/</code>.
| + | |
| − | | + | |
| − | ===CAEN 1527===
| + | |
| − | | + | |
| − | The CAEN1527 is used for the BigBite hodoscope. It is instrumented
| + | |
| − | with four 48 channel A1932 cards. These cards differ from most other
| + | |
| − | HV cards. There is a single primary power supply and 48 secondary
| + | |
| − | channels which supply the PMTs. The voltage on the primary supply can
| + | |
| − | be set to an arbitrary value. The secondary channels can only be set
| + | |
| − | to the primary voltage or less. Not all attributes are available on
| + | |
| − | both primary and secondary channels. When an attribute doesn't exist,
| + | |
| − | the GUI will show -1.0.
| + | |
| − | | + | |
| − | ===Wiener MPOD===
| + | |
| − | | + | |
| − | UVA GEM modules. Wiener MPOD systems are controlled by SNMP (Simple
| + | |
| − | Network Management Protocol). The standard EPICS
| + | |
| − | [https://groups.nscl.msu.edu/controls/files/devSnmp.html SNMP EPICS
| + | |
| − | driver] is used.
| + | |
| − | | + | |
| − | The IOC for the Wiener MPOD system runs on adaqsc. It is started by
| + | |
| − | [[#ProcServ|ProcServMgr]].
| + | |
| − | | + | |
| − | ===CAEN VME V6521 cards===
| + | |
| − | | + | |
| − | CAEN VME V6521 cards are used for the INFN GEM detectors. These cards
| + | |
| − | do not use a custom HV mainframe, just a regular VME crate. This
| + | |
| − | EPICS driver for these cards runs on the Linux Single board computer
| + | |
| − | (sbsvme27) in the VME crate with the cards. The driver software for
| + | |
| − | these cards is in <code>sbs-epics/apps/caenV65XXApp</code> and the startup script for the IOC
| + | |
| − | is <code>sbs-epics/apps/iocBoot/iocv65xx/st.cmd</code>. Slot number used in the EPICS PVs is
| + | |
| − | not the physical slot but rather the logical slot, where slot 0 is the
| + | |
| − | first card defined, slot 1 is the second card defined etc. The cards
| + | |
| − | are defined with the command
| + | |
| − | | + | |
| − | V65XXInit(Ncards, baseaddr, incaddr)
| + | |
| − | | + | |
| − | in <code>st.cmd</code>. For example, <code>V65XXInit(3, 0xa00000,
| + | |
| − | 0x100000)</code> will define "slots" 0-2 with VME addresses
| + | |
| − | <code>0xa00000, 0xb00000, and 0xc00000</code>.
| + | |
| − | | + | |
| − | ===CAEN 4527===
| + | |
| − | The CAEN 4527 is a relatively modern high voltage system. It may be used for some SBS experiments
| + | |
| − | such as GeN-RP.
| + | |
| − | | + | |
| − | The CAEN 4527 has built in EPICS. The EPICS server can be enabled in the mainframe's web interface.
| + | |
| − | The EPICS PV name prefix (e.g. "HAHV") and mainframe number are also set in this interface.
| + | |
| − | | + | |
| − | The CAEN 4527 EPICS interface does not supply separate signal names for setting and readback of setting.
| + | |
| − | For example, the PV HAHV93:00:000:V0Set is used both to set the voltage and to readback the voltage
| + | |
| − | setting. In order provide compatibility with the other EPICS drivers, the Alias4527 app is run. This
| + | |
| − | app mirrors each of writable signal names to provide a "fake" readback signal. The Alias4527 also provides a VDiff
| + | |
| − | signal for each channel for alarms.
| + | |
| − | | + | |
| − | These mirror signals are configured in the file
| + | |
| − | <code>sbs-epics/apps/Alias4725App/SY4527_crate93.substitutions</code>. (Assuming that the 4527 mainframe number is 93.)
| + | |
