Commit cb64eb7f authored by walker's avatar walker
Browse files

setup instructions for cluster

parents
.DS_Store
*png
\ No newline at end of file
# Exploration of cellpose
For setup instructions see [here](setting_up.md).
The project is work in progress ...
\ No newline at end of file
name: cellpose
channels:
- defaults
dependencies:
- _libgcc_mutex=0.1=main
- backcall=0.2.0=py_0
- blas=1.0=mkl
- ca-certificates=2020.10.14=0
- certifi=2020.11.8=py37h06a4308_0
- cloudpickle=1.6.0=py_0
- cycler=0.10.0=py37_0
- cytoolz=0.11.0=py37h7b6447c_0
- dask-core=2.30.0=py_0
- dbus=1.13.18=hb2f20db_0
- decorator=4.4.2=py_0
- expat=2.2.10=he6710b0_2
- fontconfig=2.13.0=h9420a91_0
- freetype=2.10.4=h5ab3b9f_0
- glib=2.66.1=h92f7085_0
- gst-plugins-base=1.14.0=hbbd80ab_1
- gstreamer=1.14.0=hb31296c_0
- icu=58.2=he6710b0_3
- imageio=2.9.0=py_0
- intel-openmp=2020.2=254
- ipykernel=5.3.4=py37h5ca1d4c_0
- ipython=7.19.0=py37hb070fc8_0
- ipython_genutils=0.2.0=pyhd3eb1b0_1
- jedi=0.17.2=py37_0
- jpeg=9b=h024ee3a_2
- jupyter_client=6.1.7=py_0
- jupyter_core=4.7.0=py37h06a4308_0
- kiwisolver=1.3.0=py37h2531618_0
- lcms2=2.11=h396b838_0
- ld_impl_linux-64=2.33.1=h53a641e_7
- libedit=3.1.20191231=h14c3975_1
- libffi=3.3=he6710b0_2
- libgcc-ng=9.1.0=hdf63c60_0
- libgfortran-ng=7.3.0=hdf63c60_0
- libpng=1.6.37=hbc83047_0
- libsodium=1.0.18=h7b6447c_0
- libstdcxx-ng=9.1.0=hdf63c60_0
- libtiff=4.1.0=h2733197_1
- libuuid=1.0.3=h1bed415_2
- libxcb=1.14=h7b6447c_0
- libxml2=2.9.10=hb55368b_3
- lz4-c=1.9.2=heb0550a_3
- matplotlib=3.3.2=0
- matplotlib-base=3.3.2=py37h817c723_0
- mkl=2020.2=256
- mkl-service=2.3.0=py37he904b0f_0
- mkl_fft=1.2.0=py37h23d657b_0
- mkl_random=1.1.1=py37h0573a6f_0
- ncurses=6.2=he6710b0_1
- networkx=2.5=py_0
- numpy-base=1.19.2=py37hfa32c7d_0
- olefile=0.46=py37_0
- openssl=1.1.1h=h7b6447c_0
- parso=0.7.0=py_0
- pcre=8.44=he6710b0_0
- pexpect=4.8.0=pyhd3eb1b0_3
- pickleshare=0.7.5=py37_1001
- pillow=8.0.1=py37he98fc37_0
- pip=20.3=py37h06a4308_0
- prompt-toolkit=3.0.8=py_0
- ptyprocess=0.6.0=pyhd3eb1b0_2
- pygments=2.7.2=pyhd3eb1b0_0
- pyparsing=2.4.7=py_0
- pyqt=5.9.2=py37h05f1152_2
- python=3.7.9=h7579374_0
- python-dateutil=2.8.1=py_0
- pywavelets=1.1.1=py37h7b6447c_2
- pyyaml=5.3.1=py37h7b6447c_1
- pyzmq=20.0.0=py37h2531618_1
- qt=5.9.7=h5867ecd_1
- readline=8.0=h7b6447c_0
- scikit-image=0.17.2=py37hdf5156a_0
- setuptools=50.3.1=py37h06a4308_1
- sip=4.19.8=py37hf484d3e_0
- six=1.15.0=py37h06a4308_0
- sqlite=3.33.0=h62c20be_0
- tk=8.6.10=hbc83047_0
- toolz=0.11.1=py_0
- tornado=6.0.4=py37h7b6447c_1
- traitlets=5.0.5=py_0
- wcwidth=0.2.5=py_0
- wheel=0.35.1=pyhd3eb1b0_0
- xz=5.2.5=h7b6447c_0
- yaml=0.2.5=h7b6447c_0
- zeromq=4.3.3=he6710b0_3
- zlib=1.2.11=h7b6447c_3
- zstd=1.4.5=h9ceee32_0
- pip:
- cachetools==4.1.1
- cellpose==0.5
- cffi==1.14.4
- chardet==3.0.4
- google-api-core==1.23.0
- google-auth==1.23.0
- google-cloud-core==1.4.3
- google-cloud-storage==1.33.0
- google-crc32c==1.0.0
- google-resumable-media==1.1.0
- googleapis-common-protos==1.52.0
- idna==2.10
- llvmlite==0.35.0
- mxnet-cu90==1.5.0
- natsort==7.1.0
- numba==0.52.0
- numpy==1.16.6
- opencv-python-headless==4.4.0.46
- protobuf==3.14.0
- pyasn1==0.4.8
- pyasn1-modules==0.2.8
- pycparser==2.20
- pyqtgraph==0.11.0rc0
- python-graphviz==0.8.4
- pytz==2020.4
- requests==2.25.0
- rsa==4.6
- scipy==1.5.4
- tifffile==2020.11.26
- tqdm==4.54.0
- urllib3==1.26.2
name: cellpose
dependencies:
- python>3.4,<3.8
- pip
- pyqt
- pip:
- numpy<=1.16.6,>1.14.6
- numba>=0.43.1
- scipy
- mxnet-mkl==1.5.0
- opencv-python-headless
- pyqtgraph==0.11.0rc0
- natsort
- google-cloud-storage
- tqdm
- tifffile
- cellpose
# Setting up cellpose
Author: Noreen Walker
Date: 2020-12
**This document is work in progress.**
## Overview
This file documents my efforts of setting up **cellpose with GPU support**. Installation is done with **conda** package manager.
### GPU support
For 2D data, cellpose can simply be run on the local laptop, e.g. via the GUI. For 3D a GPU is required.
Two options:
* **Cluster**: jupyter notebook or submitting a GPU job
* **LIPS workstation** (LIPS4 should have CUDA installed)
### Machine learning library: mxnet
* Cellpose uses [**mxnet**](https://mxnet.apache.org/versions/1.7.0/get_started?) and not tensorflow.
* For GPU support, the mxnet version matching your cuda version has to be installed with pip (e.g. [mxnet-cu100 for cuda10.0](https://pypi.org/project/mxnet-cu100/)).
### Resources
* general installation: https://github.com/MouseLand/cellpose
* install GPU python packages: https://github.com/MouseLand/cellpose/blob/master/notebooks/run_cellpose_GPU.ipynb
* installation detailed: https://cellpose.readthedocs.io/en/latest/installation.html
## Cluster setup
### Good to know
* jupyterhub runs `cuda9.0.176`
* This cannot be changed by the user (only Oscar can update this).
* For submitted GPU jobs we could load a cuda>=10.0 module.
* *Cellpose authors recommend however cuda>=10.0 .*
To create **one environment that can be used for both jupyterhub and job submissions I used cuda9.0.176.**
### Installation
#### Prerequisite:
* `CONDA_ENVS_PATH` and `CONDA_PKGS_DIRS` are defined and point to a project space location
* recommendation: add them to *.bashrc*
* Otherwise, conda will install everything into your small *home* directory.
* See instructions [here in TLDR section](https://git.mpi-cbg.de/scicomp/conda_and_jupyter/-/blob/master/how-to-setup-conda-environment.md)
#### Step-by-step:
Run all commands from terminal - unless python/jupyter is specifically mentioned.
1. Create a **new environment *cellpose*** with the environment file.
* `conda env create -f environment_cellpose_from_authors.yml`
* This environment file is a copy of the original file from [here](https://raw.githubusercontent.com/MouseLand/cellpose/master/environment.yml).
2. **Activate** the environment: `conda activate cellpose`
1. Add missing libraries: `conda install matplotlib scipy scikit-image`
3. Check your **cuda version**:
* From jupyter (with GPU): `!nvcc --version`
* From interactive job on GPU node:
* loading the cuda module sets the version: `module load cuda/9.0.176`
* verify also with `nvcc --version`
4. Replace the CPU **mxnet package** with a GPU version.
* `pip uninstall mxnet-mkl`
* `pip uninstall mxnet `
* `pip install mxnet-cu90=1.5.0` (For cuda9.0, adjust for other versions)
5. Install kernel for **jupyter notebook** usage:
* install kernel package: `conda install ipykernel`
* make environment visible in jupyter: </br>`python -m ipykernel install --user --name=cellpose` (details [here](https://git.mpi-cbg.de/scicomp/conda_and_jupyter/-/blob/master/how-to-setup-jupyter-kernel.md))
6. Run the **test notebook** `run_cellpose_GPU_demo.ipynb`
* From jupyterhub start a server with GPU support
* Open notebook, set *kernel to cellpose* and run it <br/><br/>
7. Optional: Create an **environment *cellposecpu*** that can run on the CPU.
* **Provide the new environment name** - otherwise *cellpose* environment will be overwritten!: <br>
`conda env create --file environment_cellpose_from_authors.yml -n cellposecpu`
* `conda activate cellposecpu`
* `conda install matplotlib scipy scikit-image`
* `conda install ipykernel`
* `python -m ipykernel install --user --name=cellposecpu`
* Run the test notebook `run_cellpose_demo.ipynb`
#### Alternative:
* To recreate the exact same *cellpose* environment that is currently used on the cluster install from the environment snapshot:
* `conda env create -f environment_cellpose_at_scf_cuda90_allpackages.yml`
* This installs mxnet-cu90.
### Usage
* **Jupyter:**
* Launch notebook with GPU from jupyter.mpi-cbg.de
* Use the cellpose kernel
* Submitted job:
* Load module: `module load cuda/9.0.176`
* Activate python environment: `conda activate cellpose`
* Launch/Run your python programm.
### Various
#### packages
* Use `python=3.7`. A test installation with 3.8 did not work.
* mxnet>1.5.0 did not work in the used environment & with `cuda9.0.176`. Use `mxnet=1.5.0`
* *For newer cuda: most likely use a newer mxnet like 1.7.0 (or don't specify the version)*
* Upgrading cellpose (Tip from [here](https://cellpose.readthedocs.io/en/latest/installation.html): upgrade without the dependencies as mxnet was manually installed):
* `pip install --no-deps cellpose --upgrade`
#### cluster
* don't use `pip install` from jupyter notebook. It defaults to the system pip, not the pip in the conda environment.
* For submitted jobs: No cuda is loaded by default: you always have to load a module.
* `nvidia-smi` does not report the currently loaded cuda module! Use `nvcc --version` instead.
* launch interactive session on GPU (4h): - very good for debugging </br>
`srun --pty --time=4:00:00 --mem-per-cpu=4000 --partition=gpu --gres=gpu:1 bash`
### TODO (possibly)
* cellpose downloads the pretrained models to */.cellpose* folder in home directory. Ca 200MB. Move to project space and create symbolic link.
### Troubleshooting
* Note: after setting up environment and running notebook on cluster I get import errors on first import try - and it works from the 2nd try on.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment