<style type="text/stylesheet"> div { text-align: left; } img[src$='#logo'] { height: 2em; } .reveal img { border: 0px !important; } </style> ## Current state of the CoLoMoTo Docker distribution #### Loïc Paulevé #### https://loicpauleve.name #### 29 May 2024, Paris  ---  Challenges: - many tools produced by the community, but difficult to install, different interfaces, ... - many of modeling papers in systems biology were not reproducible (not all parameters detailed, tools evolved, ...) ---  --- ### State in 2018  --- ### State in 2024  --- # State in 2024 - user perspective --- ### Basic usage 1. Online https://mybinder.org/v2/gh/colomoto/colomoto-docker/mybinder/latest Nothing required except a browser, and *patience* Limited computing resources + non-permanent 1. Using Python (helper script) and Docker ```bash pip install -U colomoto-docker # or python -m pip ... colomoto-docker -V latest colomoto-docker -V 2018-05-29 ``` 3. Using Docker ``` docker run -it --rm -p 8888:8888 colomoto/colomoto-docker:2024-04-01 ``` --- ### Usage tips :warning: except within the *persistent* directory, any change will be lost when closing your session **Use the --bind option** to mount a folder into the Docker ```bash colomoto-docker --bind my_notebooks/ ``` **Common pitfalls for reproducible notebooks** ● ensure that "Kernel > Restart & Run All" works! ● external resources (files, URLs, ..) ⮩ always *join model files and/or consider using zenodo* to have persistent URLs ● randomness in results (e.g., ordering of lists) ⮩ explicit sorting of results and/or condition for selecting an element in a list --- ### Available images - Images are timestamped so they can be retrieved later ``` colomoto-docker -V 2018-05-29 ``` ⮩ do not forget to indicate the timestamp to reproduce your notebooks! Today: 40 timestamped images. - Images are released monthly (if any new version of a tool appeared) tags are of the form `YYYY-MM-01` https://hub.docker.com/r/colomoto/colomoto-docker/tags - Images are hosted on mirrors + we plan to archive them on zenodo ⮩ we work on ensuring their long term storage, and re-building --- # State in 2024 - tool developer perspective --- ### How to integrate your tool in the Docker distribution https://github.com/colomoto/colomoto-docker This is possible as soon as your software - has a Python interface (or can be mixed with the Python kernel) - has a conda package for Linux - is able to interact with at least one tool already in the Docker ⮩ we require a notebook showing basic tool usage + combination with another integraed tool --- ### Step 0. Python interface If your tool is mainly command line - it is not so difficult to do. Essentially a matter of building the command line and parsing results - E.g., https://github.com/colomoto/bns-python If your tool has a GUI - in theory, we can include it. But let's not do that for now. If your tool runs on another language - Jupyter can mix kernels (R, ..) with variable passing methods - For R, there is generic Python interface E.g., [R-BoolNet](https://nbviewer.org/github/colomoto/colomoto-docker/blob/master/tutorials/R-BoolNet/Random%20BN%20generation%2C%20loading%20with%20biolqm%20or%20minibn.ipynb) --- ### Step 1. Conda packaging This is usually not so difficult. Especially if you already have a pip package. E.g., https://github.com/bnediction/mpbn/blob/master/conda/meta.yaml Can be integrated in your github workflow (e.g., automated packaging upon release). E.g., https://github.com/bnediction/mpbn For "external" packaging, we have https://github.com/colomoto/colomoto-conda (but version should be updated manually) 👋 Get in touch with us, we'll be happy to give a hand. --- ### Step 2. Try it in the Docker (see [CONTRIBUTING.md](https://github.com/colomoto/colomoto-docker/blob/master/CONTRIBUTING.md)) - Create a `Dockerfile` ``` FROM colomoto/colomoto-docker:2024-04-01 USER root RUN conda install -y yourpackage USER user ``` - Build your image: `docker build -t colomoto/colomoto-docker:local .` - Try it: `colomoto-docker -V local` --- ### Step 3. Create a demo notebook `colomoto-docker -V local --bind my_notebooks/` - Show main usage, can link to more detailed notebooks hosted elsewhere - Link with another tool. E.g., - use `biolqm` to import a model from SBML-qual - (for inference) export result to a Boolean network and use another tool to analyze it, ... - Try to keep it simple, and quick to run --- ### Step 4. Request inclusion If you are familiar with github and pull requests: - follow instructions at https://github.com/colomoto/colomoto-docker/blob/master/CONTRIBUTING.md If not - 👋 contact us --- # State in 2024 - maintainer perspective --- ### Behind the scenes See https://github.com/colomoto/colomoto-docker/blob/for-next/Dockerfile We rely extensively on GitHUB automated workflows - :robot_face: every day, there is a check for any new release of integrated tools (anaconda) - :robot_face: if so, a candidate image is built with updated packages - :robot_face: then, all the tutorials are ran: they are our unit tests - 👨‍🚒 if it fails, we are notified and check how to resolve - 👨‍💼 if all good, towards the end of the month we manually set a flag "ready for release" - :robot_face: at dawn of the 2nd of the month, candidate image is re-built, re-checked, and released --- ### Main maintenance burden: candidate image fails Core dependencies = Python, Java, usual Python packages (numpy, pandas, jupyter) - we use layers and cache of Docker images to reduce update of core dependencies - from time to time we try to upgrade the core dependencies (essentially Python version) Why tests (usually) fail: - a tool has changed its interface (API/options) → update the tutorial - conflict with dependencies → adjust version requirements 🕵️ - incompatibility with updated core dependencies: E.g. tool does no support Python 3.12 → postpone upgrade of Python, patch tool 🕵️🕵️ --- ## Discussion - need effort on "wrap-up" documentation - e.g., by feature (e.g. what are all the tools I can use to compute steady states..) - need effort to keep up with technological changes (jupyter, Docker, Python...) - not big effort (~1 day / 3 month) - enhance reproducibility of image building? - can be very tricky ; maybe not worth it - will happen: how to deal with unmaintained software? (e.g., Python incompatibility, ..) - computing ressources for dedicated mybinder servers?