Create stable conda enviroment for Single cell analysis conda on falcon

Following up from #77 (closed), the goal is now to use conda and create stable, shareable environments with different versions of tools / R that we could use for projects.

Instructions can be found here:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually

I will create two files:

• minimal, with just the packages that were manually installed, and some R packages will have to be manually installed.
• full, with everything including dependencies, but it might not work across different platforms.

conda env export --from-history > environment.R_3.5.1_essentials.minimal.yml
conda env export > environment.R_3.5.1_essentials.full.yml

added 15m of time spent at 2020-01-16

Using the full env file installation went very smoothly. R packages such as scater are missing probably because they were installed in R.

IMPORTANT: is an R module is loaded, it will take precedence over the conda R binary.

We should consider installing some R packages with conda, for instance dotcop and devtools to speed up the the env creation and reduce the manual steps.

Despite having to install many packages manually, with install.packages and BiocManager::install, there were no issues doing this inside the conda env. I rendered the qc analysis of the test dataset (organoids) and it went without issues (except the all.equal problem).

It seems like this is working.

added 2h of time spent at 2020-01-16

closed

reopened

A second attempt to create the SC stable environment, using conda to install all (or as many as possible) R packages with conda install. The idea is to avoid conflicting dependencies and keep a tight control over what is installed.

steps:

1. parse sessionInfo.txt files to obtain R packages and version used in a stable workflow.
2. install packages
3. check for packages that for some reason were not installed and
4. install the closest possible version of missing packages

Interesting discussion here: https://community.rstudio.com/t/why-not-r-via-conda/9438/4

added 1h of time spent at 2020-01-21

added 4h of time spent at 2020-01-22

I have also created a document explaining how the process works, designed both for our use and for external users. It be committed once we have decided where this will go.

added 1h 30m of time spent at 2020-01-23

@herseman the conda environments are now ready and each have been tested in a project (SC or DGE). It would be useful if you could also test it to find any potential issues early on. As far as my testing goes, it is ready for production.

To get you started on the cluster just call conda activate /projects/conda/groups/bioinfo/envs/sc_BioC3.8 for single cell or conda activate /projects/conda/groups/bioinfo/envs/dge_BioC3.8. A mode detailed description can be found in:

• DGE https://git.mpi-cbg.de/bioinfo/ngs_tools/blob/master/misc/conda_envs/how-to-setup-conda-environment.md
• SC https://git.mpi-cbg.de/scicomp/bioinfo_team/sc_tools/blob/master/misc/conda_env/how-to-setup-conda-environment.md

To set them up in your computer just use the yml files that reside in the same folders (misc/conda_envs), to create the env with the command conda env create -f /path/to/conda_environment_dge_BioC3.8.full.yml -n dge_BioC3.8. It will be installed in the default conda folder. This is has not been tested on OSX so your testing is essential.

closed

Thank you @domingue . Currently, other sc_tools users and I are using the conda_env_py37_r35_v1/conda_env_py37_r35_v1_macOS conda environments (https://git.mpi-cbg.de/scicomp/bioinfo_team/sc_tools/tree/master/misc/conda_env/yaml_files). Except for the now solved openjdk issue in the yaml files it runs fine and à la "never change a winning horse" I am a bit reluctant to change the environment ;-) Especially, because, I also already created a yaml file compatible with macOS. The conda_env_py37_r35_v1 environment is also set up on the cluster. Please feel free to use it for single cell data analysis.

Concerning the dge_BioC3.8, I am happy to give it a try and to give you some feedback. I'll let you know once I am done with the testing.

Great! One more question:

do we a protocol in place to add packages to that conda env (conda_env_py37_r35_v1)? Or we keep it as is, and if packages are needed to other projects we put them in another env?

I would suggest that we update the conda_env_py37_r35_v1 and the corresponding yaml file in case we add further scripts to the sc_tools which need packages that are not installed yet. The goal is to provide yaml files for conda environments which allow to run everything in the sc_tools repo (see sc_tools workflow usage) and by updating the existing one, we prevent redundancy. I also specified the python and R version in the name of the corresponding conda environment and as long as there is no need to switch to other versions, I would suggest that we maintain conda_env_py37_r35_v1. However, we should work with tags to indicate different sc_tools versions and this would then also be the place were we announce changes in the yaml file/conda environment.