Difference between revisions of "Off Line Analysis for transversity"
(→= Common Error Msg : conflict database) |
|||
(37 intermediate revisions by the same user not shown) | |||
Line 13: | Line 13: | ||
This is temporary site of Off Line Analysis for transversity. | This is temporary site of Off Line Analysis for transversity. | ||
+ | |||
+ | == Analysis Software == | ||
+ | |||
+ | Besides standard [http://hallaweb.jlab.org/root/index.html Hall A analyzer], Transversity analysis software contains | ||
+ | * Database sets | ||
+ | * [http://www.jlab.org/~jinhuang/BigBiteDoc/index.html BigBite Lib] | ||
+ | * RICH Lib | ||
+ | * BB Norm Analysis Lib | ||
+ | * replay scripts | ||
+ | |||
+ | The most up2date version is loaded in Hall A [http://en.wikipedia.org/wiki/Concurrent_Versions_System CVS] server. | ||
+ | |||
+ | cvs.jlab.org:/group/halla/analysis/cvs/transversity | ||
+ | |||
+ | === CVS (Concurrent Versions System) === | ||
+ | |||
+ | For those who is not very familiar with CVS. Here is a brief introduction: | ||
+ | |||
+ | From [http://en.wikipedia.org/wiki/Concurrent_Versions_SystemConcurrent_Versions_System Wikipedia:CVS]: In the field of software development, the Concurrent Versions System (CVS), also known as the Concurrent Versioning System, is a free software revision control system. Version control system software keeps track of all work and all changes in a set of files, and allows several developers (potentially widely separated in space and/or time) to collaborate. Dick Grune developed CVS in the 1980s. CVS has become popular in the open source software world and is released under the GNU General Public License. | ||
+ | |||
+ | ==== Useful Reference Links ==== | ||
+ | |||
+ | You don't need to know all. Just check what ever command you need when you need. | ||
+ | |||
+ | * [http://en.wikipedia.org/wiki/Concurrent_Versions_System Wikipedia] | ||
+ | * [http://www.slac.stanford.edu/grp/cd/soft/cvs/cvs_cheatsheet.html CVS Cheat-sheet] | ||
+ | * [http://badgertronics.com/writings/cvs/command-line.html CVS Command Line Reference Table] | ||
+ | * [http://www.idevelopment.info/data/Programming/change_management/unix_cvs/PROGRAMMING_CVS_Commands.shtml CVS Commands w/ examples] | ||
+ | |||
+ | === CVS for Transversity === | ||
+ | |||
+ | We are in a stage that multiple people is working on development of same set of software. So CVS is introduced to keep files synchronize between people. | ||
+ | * It always keep you using most up2date version of software | ||
+ | * Distribute your updates to other people | ||
+ | * Save a modification history | ||
+ | |||
+ | === HowTo + Examples === | ||
+ | |||
+ | ==== Step 1: HowTo Start ==== | ||
+ | |||
+ | It's suggested that everyone download a copy from CVS to your working computer (could be office computer or farm), and then work on your replay and development with this copy. | ||
+ | |||
+ | Notice: | ||
+ | * A pre installed version of root and hall A analyzer is required. | ||
+ | ** For farm replay, it's already set. To use the farm version, call source /w/work5602/transversity/analysis/SourceThisFirst | ||
+ | * All the files are source code. So compile is needed after download (using script "makeall") | ||
+ | |||
+ | '''Example''': Start a shell (tcsh) | ||
+ | <pre> | ||
+ | # just to test your analyzer installation | ||
+ | [jinhuang@mitnig ~]$ analyzer | ||
+ | |||
+ | # then set up the CVS environmental variables. You might want to consider save it to your ~/.tcshrc | ||
+ | [jinhuang@mitnig ~]$ setenv CVSROOT cvs.jlab.org:/group/halla/analysis/cvs | ||
+ | [jinhuang@mitnig ~]$ setenv CVS_RSH ssh | ||
+ | [jinhuang@mitnig ~]$ setenv CVSEDITOR emacs | ||
+ | |||
+ | # Download the newest version from cvs | ||
+ | [jinhuang@mitnig ~]$ cvs co transversity | ||
+ | [jinhuang@mitnig ~]$ cd transversity/ | ||
+ | |||
+ | # run makeall script to setup symbolic links and do compiles | ||
+ | [jinhuang@mitnig ~/transversity]$ makeall | ||
+ | |||
+ | # Test your new installation | ||
+ | [jinhuang@mitnig ~/transversity]$ cd replay/ | ||
+ | [jinhuang@mitnig replay]$ analyzer | ||
+ | |||
+ | rootlogon.C: Loading BigBite Library... | ||
+ | rootlogon.C: Loading BB Normalization Analysis Library... | ||
+ | rootlogon.C: Loading macro's: | ||
+ | def.h | ||
+ | HallA_style.cxx | ||
+ | AnaShowRun.C | ||
+ | libRICH.so | ||
+ | replay_det_L.C | ||
+ | replay_BB.C | ||
+ | replay_det_BB.C | ||
+ | get_bpm_pedestals.C | ||
+ | replay_phys.C | ||
+ | replay_scalar.C | ||
+ | replay_asym.C | ||
+ | |||
+ | rootlogon.C: Done! | ||
+ | |||
+ | ************************************************ | ||
+ | * * | ||
+ | * W E L C O M E to the * | ||
+ | * H A L L A C++ A N A L Y Z E R * | ||
+ | * * | ||
+ | * Release 1.5.9 Apr 22 2009 * | ||
+ | * Based on ROOT 5.22/00 Dec 17 2008 * | ||
+ | * * | ||
+ | * For information visit * | ||
+ | * http://hallaweb.jlab.org/root/ * | ||
+ | * * | ||
+ | ************************************************ | ||
+ | |||
+ | CINT/ROOT C/C++ Interpreter version 5.16.29, Jan 08, 2008 | ||
+ | Type ? for help. Commands must be C++ statements. | ||
+ | Enclose multiple statements between { }. | ||
+ | |||
+ | analyzer [0] replay_phys(4220,1000) | ||
+ | |||
+ | # then it should begin replay | ||
+ | </pre> | ||
+ | |||
+ | ==== Step 2: HowTo Update your local copy ==== | ||
+ | |||
+ | In case of anyone else updated the CVS, a email will be automatically distributed to tran_ana@jlab.org mail list. Then you should considering update your local copy of CVS. | ||
+ | |||
+ | ===== Check Which file need updating ===== | ||
+ | Following command will check which of your local file need updating base on newest CVS version | ||
+ | <pre> | ||
+ | [jinhuang@mitnig ~]$ cd transversity/ # go to your local directory containing the local copy | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs -nq update -d # if the CVS environmental variable is not set, then repeat setting commands in step 1 | ||
+ | </pre> | ||
+ | |||
+ | or | ||
+ | |||
+ | <pre> | ||
+ | [jinhuang@mitnig ~]$ cd ~ # go to the parent directory of your local directory containing the local copy | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs -nq co transversity # if the CVS environmental variable is not set, then repeat setting commands in step 1 | ||
+ | cvs server: Updating . | ||
+ | </pre> | ||
+ | |||
+ | ===== Case 1: There is no new folder/file into CVS ===== | ||
+ | |||
+ | <pre> | ||
+ | [jinhuang@mitnig ~]$ cd transversity/ # go to your local directory containing the local copy | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs update -d # if the CVS environmental variable is not set, then repeat setting commands in step 1 | ||
+ | cvs server: Updating . | ||
+ | ........ | ||
+ | ? bigbitelib/theoptics.d <-------- if a file is on your local, but not exist in CVS, it's marked as "?"; these files are not touched during cvs update | ||
+ | P makeall <-------- if a file is updated, it's marked as "P". And it is automatically updated | ||
+ | cvs server: Updating BBNormAna | ||
+ | ........ | ||
+ | cvs server: Updating replay | ||
+ | M replay/rootlogon.C <-------- if a file is modified by you, then it's marked as "M"; The modification is not be uploaded | ||
+ | ........ | ||
+ | [jinhuang@mitnig ~/transversity]$ makeall # call makeall again to compile/update the links | ||
+ | </pre> | ||
+ | |||
+ | ===== Case 2: There are newly added folder or file in CVS ===== | ||
+ | |||
+ | <pre> | ||
+ | [jinhuang@mitnig ~]$ cd ~ # go to the parent directory of your local directory containing the local copy | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs co transversity # if the CVS environmental variable is not set, then repeat setting commands in step 1 | ||
+ | cvs server: Updating . | ||
+ | ........ | ||
+ | ? bigbitelib/theoptics.d <-------- if a file is on your local, but not exist in CVS, it's marked as "?"; these files are not touched during cvs update | ||
+ | P makeall <-------- if a file is updated, it's marked as "P". And it is automatically updated | ||
+ | cvs server: Updating BBNormAna | ||
+ | ........ | ||
+ | cvs server: Updating replay | ||
+ | M replay/rootlogon.C <-------- if a file is modified by you, then it's marked as "M"; The modification is not be uploaded | ||
+ | ........ | ||
+ | [jinhuang@mitnig ~/transversity]$ makeall # call makeall again to compile/update the links | ||
+ | </pre> | ||
+ | |||
+ | ==== Step 3: HowTo Upload a file modified by you ==== | ||
+ | |||
+ | Say you modified a new version of some files and want to share it with everyone. Then please follow this step to upload the new version to CVS. | ||
+ | |||
+ | <pre> | ||
+ | [jinhuang@mitnig ~]$ cd transversity/ # go to your local directory containing the local copy | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs update # if the CVS environmental variable is not set, then repeat setting commands in step 1 | ||
+ | cvs server: Updating . | ||
+ | ........ | ||
+ | M replay/rootlogon.C <-------- if a file is modified by you, then it's marked as "M"; The modification is not be uploaded | ||
+ | ........ | ||
+ | # Now it's clear which file is modified, then upload by | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs ci replay/rootlogon.C # include only file/file list you need to upload. "cvs ci" (without file list) would upload all modified files. | ||
+ | # A emacs will pop up for you to enter a log message. | ||
+ | # This message will be saved as description to this modification, as well as send to trans_ana@jlab.org to remind others to download your new version | ||
+ | # | ||
+ | # To add new files, use "cvs add" first before "cvs ci", like | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs add <a new file> | ||
+ | [jinhuang@mitnig ~/transversity]$ cvs ci <a new file> | ||
+ | </pre> | ||
+ | |||
+ | ==== Step 4: Find more command you need online ==== | ||
+ | |||
+ | See [[Off_Line_Analysis_for_transversity#Useful_Reference_Links]] | ||
+ | |||
+ | === Common Error Msg : conflict database === | ||
+ | |||
+ | There is a common error message like below during a cvs update: | ||
+ | |||
+ | cvs server: conflict: DB/20090201/db_BB.mwdc.dat is modified but no longer in the repository | ||
+ | |||
+ | This is caused by incompatibility of CVS about symbolic links. You can ignore this message or clear it by removing the problematic link and cvs update again: | ||
+ | |||
+ | <pre> | ||
+ | [jinhuang@mitnig ~/transversity] ls -l DB/20090201/db_BB.mwdc.dat # just to make sure it's an automatic symbolic link | ||
+ | [jinhuang@mitnig ~/transversity] rm -fv DB/20090201/db_BB.mwdc.dat | ||
+ | [jinhuang@mitnig ~/transversity] cvs update | ||
+ | [jinhuang@mitnig ~/transversity] makeall | ||
+ | </pre> | ||
== Farm Replay == | == Farm Replay == | ||
=== Structure === | === Structure === | ||
+ | |||
+ | On farm file system, there are 3 disk: | ||
+ | * disk1 : Our main script disk. | ||
+ | * disk2 : Shared with E08007, mostly full | ||
+ | * disk3 : 2TB disk for large root files | ||
+ | |||
+ | THe main analysis directory is ''/work/halla/transversity/disk1/analysis/'', which includes various of components: | ||
+ | * replay.sh = SourceMeFirst : script setting up environmental pathes. Configured for CSH | ||
+ | * replay : replay directory holding replay scripts/configure files | ||
+ | * farmreplay : specialized folder generating batch farm replay scripts | ||
+ | * DB : database | ||
+ | * bigbitelib : [http://www.jlab.org/~jinhuang/BigBiteDoc/ Bigbite library] | ||
+ | * RICHlib : rich lib | ||
+ | * BBNormAna : Normalization analysis lib | ||
+ | * onlineGUI : [http://www.jlab.org/~moffit/onlineGUI/ Online GUI] | ||
+ | |||
+ | analysis/ directorsis, which are not included in CVS, could be copied to your own location to be customized. | ||
+ | |||
+ | Also a copy of '''root''' and '''analyzer''' is compiled at /work/halla/transversity/disk1/. Calling analysis/replay.sh would set your root and analyzer to this version. | ||
=== HowTo do a quick test replay === | === HowTo do a quick test replay === | ||
Line 37: | Line 255: | ||
<pre> | <pre> | ||
ssh ifarml5 # or any computer seeing working disk | ssh ifarml5 # or any computer seeing working disk | ||
− | + | mkdir /work/halla/transversity/disk1/<your name> | |
cd /work/halla/transversity/disk1/<your name> | cd /work/halla/transversity/disk1/<your name> | ||
− | + | # then set up the CVS environmental variables. You might want to consider save it to your ~/.tcshrc | |
− | + | setenv CVSROOT cvs.jlab.org:/group/halla/analysis/cvs | |
+ | setenv CVS_RSH ssh | ||
+ | setenv CVSEDITOR emacs | ||
+ | |||
+ | cvs co transversity | ||
+ | cd transversity | ||
+ | makeall | ||
− | cp -rv /work/halla/transversity/disk1/analysis/replay ./ | + | # add additional files for batch farm running |
+ | cp -rv /work/halla/transversity/disk1/analysis/replay ./ # edit this file to redirect DB_DIR to your database dir | ||
cp -rv /work/halla/transversity/disk1/analysis/farmreplay ./ | cp -rv /work/halla/transversity/disk1/analysis/farmreplay ./ | ||
cp -rv /work/halla/transversity/disk1/analysis/replay.sh ./ | cp -rv /work/halla/transversity/disk1/analysis/replay.sh ./ | ||
Line 52: | Line 277: | ||
Now you have the basic structure: | Now you have the basic structure: | ||
− | * To test replay | + | * To test replay in your new folder, follow [[Off Line Analysis for transversity#HowTo build your own replay directory]] (need to source replay.sh) |
* To do '''batch farm replay''', follow example below: | * To do '''batch farm replay''', follow example below: | ||
*# enter directory ''farmreplay'' | *# enter directory ''farmreplay'' | ||
Line 68: | Line 293: | ||
=== HowTO Customize Analyzer / Libs / Database === | === HowTO Customize Analyzer / Libs / Database === | ||
− | + | * Please refer to [[Off_Line_Analysis_for_transversity#CVS_for_Transversity]] to download a copy of analysis software to your own folder. | |
− | * | + | * analysis/farmreplay/ also contain good example for your batch farm replay script. |
− | + | * Our new [https://wiki.jlab.org/cc/external/wiki/index.php/Batch_Job_Command_File farm project name] is e06010 | |
− | * | + | |
− | + | ||
− | + | ||
− | + | ||
− | + | === HowTO Check the replay logs === | |
− | + | ||
− | * | + | Xin invented a set of tools to go through log file of farm replay and report warnings and errors. They are in CVS transversity/log_check |
+ | |||
+ | *The first file "checklog_err" is used to check error message. The results is at the error.dat. | ||
+ | *The secon file "checklog_warning" is used to check the warning message. The results is at the warning.dat | ||
+ | *The third file "checklog_file" is used to check the number of raw data file replayed. The problem is that during farm replay, the second or third raw data file may be deleted when the first file is being replayed. | ||
== General Rule for Farm Disk Usage == | == General Rule for Farm Disk Usage == | ||
+ | |||
+ | All transversity farm disks are shared with other hall A experiment. The total space is limited. In the case that the disk is full, old file will be deleted by Hall A manager. Therefore, we need to avoid heavy disk usage by following proper rules: | ||
+ | |||
+ | On farm file system, there are 3 disks mounted at ''/work/halla/transversity/'': | ||
+ | |||
+ | === disk1, 800GB === | ||
+ | |||
+ | Disk1 is our main script disk. Please make a directory with your user name. | ||
+ | |||
+ | It could be used to store small files. Please avoid large data. Please always keep a decent percentage of free space. | ||
+ | |||
+ | It's shared with GEN, Pi0. | ||
+ | |||
+ | === disk2, 800GB === | ||
+ | |||
+ | Shared with E08007, mostly full. Could be used to buffer test files. | ||
+ | |||
+ | === disk3, 1-2TB === | ||
+ | |||
+ | TB disk for large root files buffering. Quota is controlled by Dr. Robot Michaels. If the disk is full, old files will be deleted! | ||
+ | |||
+ | * Please make a directory with your user name to separate your storage space | ||
+ | * Please keep your root file as small as possible | ||
+ | * Please delete unused root files as soon as possible | ||
+ | * ''TmpROOTfiles'' which is linked to common replay directory will be periodically cleaned | ||
+ | * ''ProductionROOTfiles'' is for farm production, performed and managed by collaboration. | ||
+ | |||
+ | === HowTo Control your root file size === | ||
+ | |||
+ | By controlling which variable to output and apply proper cut, you can not only limit the size of root file, but also speed up the replay. | ||
+ | * Output define file: Controlling which variable to write. [http://hallaweb.jlab.org/root/doc/output.html see description here] | ||
+ | * Cut File : Controlling which event to process and output. [http://hallaweb.jlab.org/root/doc/test-guide.html see description here] |
Latest revision as of 14:52, 6 July 2009
This is temporary site of Off Line Analysis for transversity.
Contents
- 1 Analysis Software
- 2 Farm Replay
- 3 General Rule for Farm Disk Usage
Analysis Software
Besides standard Hall A analyzer, Transversity analysis software contains
- Database sets
- BigBite Lib
- RICH Lib
- BB Norm Analysis Lib
- replay scripts
The most up2date version is loaded in Hall A CVS server.
cvs.jlab.org:/group/halla/analysis/cvs/transversity
CVS (Concurrent Versions System)
For those who is not very familiar with CVS. Here is a brief introduction:
From Wikipedia:CVS: In the field of software development, the Concurrent Versions System (CVS), also known as the Concurrent Versioning System, is a free software revision control system. Version control system software keeps track of all work and all changes in a set of files, and allows several developers (potentially widely separated in space and/or time) to collaborate. Dick Grune developed CVS in the 1980s. CVS has become popular in the open source software world and is released under the GNU General Public License.
Useful Reference Links
You don't need to know all. Just check what ever command you need when you need.
CVS for Transversity
We are in a stage that multiple people is working on development of same set of software. So CVS is introduced to keep files synchronize between people.
- It always keep you using most up2date version of software
- Distribute your updates to other people
- Save a modification history
HowTo + Examples
Step 1: HowTo Start
It's suggested that everyone download a copy from CVS to your working computer (could be office computer or farm), and then work on your replay and development with this copy.
Notice:
- A pre installed version of root and hall A analyzer is required.
- For farm replay, it's already set. To use the farm version, call source /w/work5602/transversity/analysis/SourceThisFirst
- All the files are source code. So compile is needed after download (using script "makeall")
Example: Start a shell (tcsh)
# just to test your analyzer installation [jinhuang@mitnig ~]$ analyzer # then set up the CVS environmental variables. You might want to consider save it to your ~/.tcshrc [jinhuang@mitnig ~]$ setenv CVSROOT cvs.jlab.org:/group/halla/analysis/cvs [jinhuang@mitnig ~]$ setenv CVS_RSH ssh [jinhuang@mitnig ~]$ setenv CVSEDITOR emacs # Download the newest version from cvs [jinhuang@mitnig ~]$ cvs co transversity [jinhuang@mitnig ~]$ cd transversity/ # run makeall script to setup symbolic links and do compiles [jinhuang@mitnig ~/transversity]$ makeall # Test your new installation [jinhuang@mitnig ~/transversity]$ cd replay/ [jinhuang@mitnig replay]$ analyzer rootlogon.C: Loading BigBite Library... rootlogon.C: Loading BB Normalization Analysis Library... rootlogon.C: Loading macro's: def.h HallA_style.cxx AnaShowRun.C libRICH.so replay_det_L.C replay_BB.C replay_det_BB.C get_bpm_pedestals.C replay_phys.C replay_scalar.C replay_asym.C rootlogon.C: Done! ************************************************ * * * W E L C O M E to the * * H A L L A C++ A N A L Y Z E R * * * * Release 1.5.9 Apr 22 2009 * * Based on ROOT 5.22/00 Dec 17 2008 * * * * For information visit * * http://hallaweb.jlab.org/root/ * * * ************************************************ CINT/ROOT C/C++ Interpreter version 5.16.29, Jan 08, 2008 Type ? for help. Commands must be C++ statements. Enclose multiple statements between { }. analyzer [0] replay_phys(4220,1000) # then it should begin replay
Step 2: HowTo Update your local copy
In case of anyone else updated the CVS, a email will be automatically distributed to tran_ana@jlab.org mail list. Then you should considering update your local copy of CVS.
Check Which file need updating
Following command will check which of your local file need updating base on newest CVS version
[jinhuang@mitnig ~]$ cd transversity/ # go to your local directory containing the local copy [jinhuang@mitnig ~/transversity]$ cvs -nq update -d # if the CVS environmental variable is not set, then repeat setting commands in step 1
or
[jinhuang@mitnig ~]$ cd ~ # go to the parent directory of your local directory containing the local copy [jinhuang@mitnig ~/transversity]$ cvs -nq co transversity # if the CVS environmental variable is not set, then repeat setting commands in step 1 cvs server: Updating .
Case 1: There is no new folder/file into CVS
[jinhuang@mitnig ~]$ cd transversity/ # go to your local directory containing the local copy [jinhuang@mitnig ~/transversity]$ cvs update -d # if the CVS environmental variable is not set, then repeat setting commands in step 1 cvs server: Updating . ........ ? bigbitelib/theoptics.d <-------- if a file is on your local, but not exist in CVS, it's marked as "?"; these files are not touched during cvs update P makeall <-------- if a file is updated, it's marked as "P". And it is automatically updated cvs server: Updating BBNormAna ........ cvs server: Updating replay M replay/rootlogon.C <-------- if a file is modified by you, then it's marked as "M"; The modification is not be uploaded ........ [jinhuang@mitnig ~/transversity]$ makeall # call makeall again to compile/update the links
Case 2: There are newly added folder or file in CVS
[jinhuang@mitnig ~]$ cd ~ # go to the parent directory of your local directory containing the local copy [jinhuang@mitnig ~/transversity]$ cvs co transversity # if the CVS environmental variable is not set, then repeat setting commands in step 1 cvs server: Updating . ........ ? bigbitelib/theoptics.d <-------- if a file is on your local, but not exist in CVS, it's marked as "?"; these files are not touched during cvs update P makeall <-------- if a file is updated, it's marked as "P". And it is automatically updated cvs server: Updating BBNormAna ........ cvs server: Updating replay M replay/rootlogon.C <-------- if a file is modified by you, then it's marked as "M"; The modification is not be uploaded ........ [jinhuang@mitnig ~/transversity]$ makeall # call makeall again to compile/update the links
Step 3: HowTo Upload a file modified by you
Say you modified a new version of some files and want to share it with everyone. Then please follow this step to upload the new version to CVS.
[jinhuang@mitnig ~]$ cd transversity/ # go to your local directory containing the local copy [jinhuang@mitnig ~/transversity]$ cvs update # if the CVS environmental variable is not set, then repeat setting commands in step 1 cvs server: Updating . ........ M replay/rootlogon.C <-------- if a file is modified by you, then it's marked as "M"; The modification is not be uploaded ........ # Now it's clear which file is modified, then upload by [jinhuang@mitnig ~/transversity]$ cvs ci replay/rootlogon.C # include only file/file list you need to upload. "cvs ci" (without file list) would upload all modified files. # A emacs will pop up for you to enter a log message. # This message will be saved as description to this modification, as well as send to trans_ana@jlab.org to remind others to download your new version # # To add new files, use "cvs add" first before "cvs ci", like [jinhuang@mitnig ~/transversity]$ cvs add <a new file> [jinhuang@mitnig ~/transversity]$ cvs ci <a new file>
Step 4: Find more command you need online
See Off_Line_Analysis_for_transversity#Useful_Reference_Links
Common Error Msg : conflict database
There is a common error message like below during a cvs update:
cvs server: conflict: DB/20090201/db_BB.mwdc.dat is modified but no longer in the repository
This is caused by incompatibility of CVS about symbolic links. You can ignore this message or clear it by removing the problematic link and cvs update again:
[jinhuang@mitnig ~/transversity] ls -l DB/20090201/db_BB.mwdc.dat # just to make sure it's an automatic symbolic link [jinhuang@mitnig ~/transversity] rm -fv DB/20090201/db_BB.mwdc.dat [jinhuang@mitnig ~/transversity] cvs update [jinhuang@mitnig ~/transversity] makeall
Farm Replay
Structure
On farm file system, there are 3 disk:
- disk1 : Our main script disk.
- disk2 : Shared with E08007, mostly full
- disk3 : 2TB disk for large root files
THe main analysis directory is /work/halla/transversity/disk1/analysis/, which includes various of components:
- replay.sh = SourceMeFirst : script setting up environmental pathes. Configured for CSH
- replay : replay directory holding replay scripts/configure files
- farmreplay : specialized folder generating batch farm replay scripts
- DB : database
- bigbitelib : Bigbite library
- RICHlib : rich lib
- BBNormAna : Normalization analysis lib
- onlineGUI : Online GUI
analysis/ directorsis, which are not included in CVS, could be copied to your own location to be customized.
Also a copy of root and analyzer is compiled at /work/halla/transversity/disk1/. Calling analysis/replay.sh would set your root and analyzer to this version.
HowTo do a quick test replay
One can directly use the common replay directory to do quick tests.
ssh ifarml5 # or any computer seeing working disk cd /work/halla/transversity/disk1/analysis source replay.sh cd replay analyzer replay_phys(4029,100000); //just like online replay
HowTo build your own batch farm replay directory
Build a directory for your own replay. You can change replay code and control the rootfile storage. And it will be capable of replaying on both ifarm and batch farm.
ssh ifarml5 # or any computer seeing working disk mkdir /work/halla/transversity/disk1/<your name> cd /work/halla/transversity/disk1/<your name> # then set up the CVS environmental variables. You might want to consider save it to your ~/.tcshrc setenv CVSROOT cvs.jlab.org:/group/halla/analysis/cvs setenv CVS_RSH ssh setenv CVSEDITOR emacs cvs co transversity cd transversity makeall # add additional files for batch farm running cp -rv /work/halla/transversity/disk1/analysis/replay ./ # edit this file to redirect DB_DIR to your database dir cp -rv /work/halla/transversity/disk1/analysis/farmreplay ./ cp -rv /work/halla/transversity/disk1/analysis/replay.sh ./ mkdir /work/halla/transversity/disk3/<your root file folder> ln -svf /work/halla/transversity/disk3/<your root file folder> ROOTfiles
Now you have the basic structure:
- To test replay in your new folder, follow Off Line Analysis for transversity#HowTo build your own replay directory (need to source replay.sh)
- To do batch farm replay, follow example below:
- enter directory farmreplay
- edit construct.pl, modify following sections:
- $neve : the event number to replay
- @runnumber : list of run to replay
- $ReplayType : name of replay script
- $current : path of your analysis directory
- run construct.pl; it will generate
- in_run# : the script for batch running for each run; double check it carefully
- sub_run# : Batch Job Command File for each run; double check it carefully
- (Optional), test in_run# scripts by copying it to a tmp directory and run it.
- jsub sub_run# will run the replay in batch farm
HowTO Customize Analyzer / Libs / Database
- Please refer to Off_Line_Analysis_for_transversity#CVS_for_Transversity to download a copy of analysis software to your own folder.
- analysis/farmreplay/ also contain good example for your batch farm replay script.
- Our new farm project name is e06010
HowTO Check the replay logs
Xin invented a set of tools to go through log file of farm replay and report warnings and errors. They are in CVS transversity/log_check
- The first file "checklog_err" is used to check error message. The results is at the error.dat.
- The secon file "checklog_warning" is used to check the warning message. The results is at the warning.dat
- The third file "checklog_file" is used to check the number of raw data file replayed. The problem is that during farm replay, the second or third raw data file may be deleted when the first file is being replayed.
General Rule for Farm Disk Usage
All transversity farm disks are shared with other hall A experiment. The total space is limited. In the case that the disk is full, old file will be deleted by Hall A manager. Therefore, we need to avoid heavy disk usage by following proper rules:
On farm file system, there are 3 disks mounted at /work/halla/transversity/:
disk1, 800GB
Disk1 is our main script disk. Please make a directory with your user name.
It could be used to store small files. Please avoid large data. Please always keep a decent percentage of free space.
It's shared with GEN, Pi0.
disk2, 800GB
Shared with E08007, mostly full. Could be used to buffer test files.
disk3, 1-2TB
TB disk for large root files buffering. Quota is controlled by Dr. Robot Michaels. If the disk is full, old files will be deleted!
- Please make a directory with your user name to separate your storage space
- Please keep your root file as small as possible
- Please delete unused root files as soon as possible
- TmpROOTfiles which is linked to common replay directory will be periodically cleaned
- ProductionROOTfiles is for farm production, performed and managed by collaboration.
HowTo Control your root file size
By controlling which variable to output and apply proper cut, you can not only limit the size of root file, but also speed up the replay.
- Output define file: Controlling which variable to write. see description here
- Cut File : Controlling which event to process and output. see description here