View source for Note about container

questions and comments should go to Zhiwen Zhao zwzhao at jlab.org

= general =

Users can run docker or singularity/apptainer container on any machine with no installation needed and obtain consistent result

docker images are common used for container because both docker and singularity can load docker image.

on linux, singularity is preferred because docker need root access or sudo and not supported at most machines in jlab.

on windows/mac, singularity can be used also https://apptainer.org/docs/admin/main/installation.html#installation-on-windows-or-mac, before trying docker

To use graphic software inside a container as it is, instead of piping through host, the way is to use vnc server inside container, then connect to it by vncviewer. 
you may get vncviewer by "yum install tigervnc" or download a standalone version at https://www.realvnc.com/en/connect/download/viewer/linux/

= singularity =
* general 
** website new https://apptainer.org/docs/admin/main/installation.html https://www.sylabs.io/docs/ old http://singularity.lbl.gov  github https://github.com/sylabs/singularity/releases
** hub https://singularity-hub.org
** talk "Singularity: Containers for Science, Reproducibility, and HPC"  https://youtu.be/DA87Ba2dpNM

* warning
* installation
** always try to get the latest singularity
*** if you are running singularity-2.4.x on a centos7.2 host like jlab ifarm and farm, you could have kernel panic if you use ROOT TMD5 and share your host dir. 
** On redhat-kind linux, build rpm first and install it 
*** instruction for 2.x [http://singularity.lbl.gov/install-linux installation instruction]. at least "squashfs-tools,automake,rpm-build,libtool" are needed to build it.
*** instruction for 3.x [https://sylabs.io/guides/3.4/user-guide/installation.html#install-on-linux]
** on centos/rhel, enable epel repo and you can find latest rpms by search singularity
** on fedora, you can find latest rpms by search singularity
** On windows and Mac, the official way is just running a small linux virtual machine with virtualbox, but the virtual machine is not good for graphic application. The better alternative is to use a good but not so small linux virtual machine. If you have one already, use it. If not, see below to download an existing one.

* singularity at jlab ifarm and farm
** jlab farm and ifarm has singularity installed at /apps/singularity/, there could be several version installed
** it will auto bind "/u /w /group /work /cache /volatile /scratch" into any container by default as controlled by "singularity.conf". for example, "/usr/local/singularity/singularity-someverson/etc/singularity/singularity.conf"
** load by "module load singularity", stop by "module rm singularity" (2.6.1 current default, 3.2.1 is for test,as 2019/06)
** Before you need do any pull from outside jlab
*** if somehow you can't access internet from ifarm, "setenv http_proxy http://jprox.jlab.org:8082" and "setenv https_proxy http://jprox.jlab.org:8082" to use jlab proxy
*** '''"setenv SINGULARITY_CACHEDIR /scratch/$USER" "setenv SINGULARITY_TMPDIR /scratch/$USER" to change cache and tmp dir from default ~/.singularity and /tmp. MUST do at jlab ifarm with very limited space at home and tmp'''
** If Xwindow forwarding doesn't work for you somehow, try this
***on ifarm, echo $DISPLAY                 (host display would should something similar to "129.57.70.22:34.0", set it exactly same later inside container)
*** inside container "setenv DISPLAY 129.57.70.22:34.0" or "export DISPLAY=129.57.70.22:34.0" (replace the actual port with what you have)
*** inside container "xterm"   (test if you can pass X11 application from inside container through ssh to your local machine)

* vnc inside singularity 
** run vncserver inside a singularity container will start a vnc server which can be accessed from both the host and the container by a vncviewer

* test singularity 
** singularity pull docker://godlovedc/lolcow
** singularity run lolcow.simg
** setenv SINGULARITY_CACHEDIR /scratch/$USER
** setenv SINGULARITY_TMPDIR /scratch/$USER
** setenv PYTHONHTTPSVERIFY 0 (sometime needed to bypass singularity hub certificate check)
** singularity pull shub://GodloveD/lolcow
** singularity run GodloveD-lolcow-master-latest.simg

* note
** when singularity pull docker image, it pull and build singularity image right away. It can fail sometimes because docker image are made as root user. you have to singularity pull with sudo then

=  docker =
* general
** check space by "docker system df" and free space with "docker system prune", refer to [https://stackoverflow.com/questions/44288901/how-to-force-docker-to-release-storage-space-after-manual-delete-of-file-in-volu][https://docs.docker.com/config/pruning]
** remove tag by "docker rmi tagname"
** remove image by "docker rmi imageID"
** "docker image ls --digests" and "docker image inspect imageID"
* at linux
** [https://techoverflow.net/2017/03/01/solving-docker-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket/ Solving Docker permission denied while trying to connect to the Docker daemon socket]
** install the new version like 18.* from docker repo, not old 1.* from centos extras repo or ubuntu repo (docker version format changed in 2017/03 [https://nickjanetakis.com/blog/docker-community-edition-vs-enterprise-edition-and-their-release-cycle])
** Do all post installation steps to add user to docker group to avoid sudo and auto start docker daemon, with instruction at https://docs.docker.com/install/linux/linux-postinstall/, then run "sudo docker run hello-world" to test
** use graphic in linux container with host's Xwindows like this "sudo docker run -it --rm -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v ${PWD}:$PWD --ipc=host container_name" and it share your current dir on host inside container
** use graphic in linux container with container's Xwindows
*** install [https://github.com/mviereck/x11docker x11docker]
**** wget https://raw.githubusercontent.com/mviereck/x11docker/master/x11docker -O /tmp/x11docker
****    sudo bash /tmp/x11docker --update
****    rm /tmp/x11docker
****    install xorg-x11-server-Xephyr by yum or xserver-xephyr by apt-get
*** make sure the container image has fluxbox and xterm installed
*** x11docker --desktop --sharedir $PWD container_name fluxbox  
*** right click on the fluxbox window and open xterm
** if your host is within jlab network but not jlab level1 or level2 linux, mostly likely you will encounter error like "x509: certificate signed by unknown authority" because jlab gateway hijack site certificate with its own. We can solve this by mimic a jlab maintained system as follows
*** do "yum install jlabca", if it fixed the problem, ignore steps below
*** copy all files under /etc/pki/ca-trust/source/anchors/ from a jlab system like ifarm to your system
*** run "update-ca-trust","service docker stop","service docker start" as root
*** refer to https://github.com/moby/moby/issues/8849 for some discussion
** images are at /var/lib/docker/overlay2
* at windows
** docker for windows requires latest window 10 pro 64bit and hyper-V on relative new CPU. docker will enable hyper-V which is disabled by default. Then your virtualbox and maybe vmware will stop working by crashing your windows, refer to https://docs.docker.com/docker-for-windows/install/#what-to-know-before-you-install
** to use a linux container with xwindows application like xterm, you need to have a xwindow server installed, you can install Xming or VcXsrv, then run XLaunch with display as 0 and "No access control" checked. Finally run "docker run -it -e DISPLAY=your_host_ip:0.0 container_name". you may find your_host_ip by "ifconfig.exe"
** images as one file at "C:\Users\Public\Documents\Hyper-V\Virtual hard disks\"
* at Mac
** need OSX Yosemite 10.10.3 or above
** There is a native Mac version of Docker. The Linux version can be run in virtualbox but this is not necessary.
** You do not need to use sudo to run docker.
** to use a linux container with xwindows application like xterm, you need to have XQuartz installed. you need to turn on "allow connection from network client" and turn off "authenticate connection" in XQuartz preferences security setting, and enable x11 access by run "xhost +your_host_ip" and set run "docker run -it -e DISPLAY=your_host_ip:0 container_name". you can find your_host_ip by "ifconfig en0 | grep inet | awk '$1=="inet" {print $2}'" or "ifconfig en1 | grep inet | awk '$1=="inet" {print $2}'"
** "Unrecognized OpenGL version" error is because Indirect GLX was disabled by default in Xorg 1.17, starting from XQuartz version 2.7.9, to enable indirect GLX and to allow remote visualization you will have to issue at the terminal command prompt of your mac: "defaults write org.macosforge.xquartz.X11 enable_iglx -bool true", refer to https://www.hoffman2.idre.ucla.edu/access/x11_forwarding/
** another instruction about using xwindows application is to use scocat https://www.cnblogs.com/noluye/p/11405358.html in Chinese

= a good linux virtual machine to run singularity and docker =
* a good linux virtual machine to run singularity and docker, centos7 made with virtualbox
** download and install virtualbox https://www.virtualbox.org/wiki/Downloads
** It's nice, but not a must, to run singularity image in shared folder because it keeps the virtual machine size small. Vmware's shared folder doesn't work for this, but virtualbox does
** download the machine at http://webhome.phy.duke.edu/~zz81/vm/CentOS7_20180303.ova (a centos7 64bit with latest update on 20180303 and singularity-2.4.2 and latest docker installed)
** import the virtual machine into virtualbox refer to [https://askubuntu.com/questions/588426/how-to-export-and-import-virtualbox-vm-images 
** boot up the linux virtual machine and login with user name "user" without password, just you know root password is "111111"
** use the installed singularity and docker, or update docker by yum or singularity by compile from source if you want
** set up a shared folder with name "share" in the machine setting and put any singularity images into the shared folder on host
** mount the shared folder "sudo mount -t vboxsf -o uid=$uid,gid=$gid share share" and use /home/user/share" as your working dir